[2025-01-08 00:50:46.286349] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=105][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.296099] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] clock out of order(cur_ts=1736326246296097, cur_ts_=1736326246275168, delta=20929) [2025-01-08 00:50:46.313611] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:46.321291] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.321326] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.331876] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58301][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD83-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326246427884, keepalive_ts:-1}) [2025-01-08 00:50:46.331928] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58301][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD83-0-0] [lt=49] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326246427884, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:46.331990] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=96, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:46.335276] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8289-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:46.335312] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8289-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4771, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:46.335486] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2335, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.342577] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=121] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=2070, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:46.342628] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=50] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10700) [2025-01-08 00:50:46.342652] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=157, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:46.342677] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=286] [PALF STAT FLUSH CB COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=35, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:46.344141] INFO [LIB] stat (utility.h:1210) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B07-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=7, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:46.344177] INFO [LIB] stat (utility.h:1210) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B07-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=3441, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:46.344190] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B07-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=2340, avg_log_submit_to_slide_time=3331) [2025-01-08 00:50:46.344371] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10700, hit_cnt=19, read_cnt=19, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.344467] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326245690224003, v:0}, new_max_version={val:1736326245690224003, v:0}, last_min_version={val:1736326245477976003, v:0}, last_max_version={val:1736326245477976003, v:0}, total_time_us=5169, delta_us=654241) [2025-01-08 00:50:46.344496] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326245690224003, v:0}, min_version={val:1736326245690224003, v:0}, max_version={val:1736326245690224003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:46.345645] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC366D-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:46.345723] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC366D-0-0] [lt=68] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:46.351692] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=44] gc stale ls task succ [2025-01-08 00:50:46.354018] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:46.365934] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43AB-0-0] [lt=16] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:46.382958] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=20, avg_log_batch_cnt=1, total_group_log_size=11289, avg_group_log_size=564) [2025-01-08 00:50:46.382997] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11289) [2025-01-08 00:50:46.386559] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.388420] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=109] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:46.388451] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-01-08 00:50:46.388461] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:46.388475] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:46.388489] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193750302122, flush_io_finished_ret_:0, flush_io_finished_round_:1905, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:46.388950] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=83] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:46.388973] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-01-08 00:50:46.388981] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:46.388990] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:46.389001] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193750302649, flush_io_finished_ret_:0, flush_io_finished_round_:3733, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:46.400087] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=45] ts response task statistics(alloc_count=59310, free_count=29654) [2025-01-08 00:50:46.405866] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1830, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:46.406069] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=199] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5289) [2025-01-08 00:50:46.406095] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=275, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:46.406131] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:46.407447] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A5F-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:46.407471] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A5F-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3546, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:46.407482] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A5F-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=99, avg_log_submit_to_flush_time=2312, avg_log_submit_to_slide_time=3447) [2025-01-08 00:50:46.407589] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5289, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.435568] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC828C-0-0] [lt=46] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=3098, avg_log_submit_to_slide_time=4543) [2025-01-08 00:50:46.442966] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [50416][T1002_TimerWK1][T1001][YB42C0A89F89-00062B2D1D185376-0-0] [lt=37] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326246437667, can_tenant_elr:true}) [2025-01-08 00:50:46.443977] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:46.444692] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=38] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:46.458863] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=656101376, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=417640448, wash_ratio=6, tenant_id=1001) [2025-01-08 00:50:46.458984] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=75] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:50:46.459005] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2785, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:46.459077] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1112293376, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2108932096, wash_ratio=6, tenant_id=1002) [2025-01-08 00:50:46.459133] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:50:46.459147] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=13] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:46.459292] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.459331] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.460675] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=211] PNIO [ratelimit] time: 1736326246460671, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007606, add_bytes: 0 [2025-01-08 00:50:46.467652] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1157, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:46.467708] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:46.467735] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=86, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:46.467805] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:46.486712] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.514667] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=84] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:46.519108] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.532446] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=77, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:46.532494] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:46.532516] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:46.532531] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:46.534059] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1742, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:46.534106] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=175, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:46.534147] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:46.536235] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC828F-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:46.536274] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC828F-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4306, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:46.541307] INFO pn_ratelimit (group.c:653) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=48] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:50:46.542020] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.542078] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58135][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB183-0-0] [lt=14] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:50:46.542105] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58135][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB183-0-0] [lt=20] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:46.542133] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58135][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB183-0-0] [lt=25] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:50:46.544299] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.550502] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1568, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:46.550545] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:46.550517] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=66551808) [2025-01-08 00:50:46.550576] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=86, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:46.550652] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:46.550823] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get_min_unreplayed_log_info(lsn={lsn:2170041}, scn={val:1736326245982947005, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326246469179, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2170041}, next_to_submit_scn_:{val:1736326245982947005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2169919}, end_lsn:{lsn:2170041}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2170041}, readable_log_tail:{lsn:2170041}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64934727, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326245982947004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326245982947004, v:0}, data_checksum:3348837161, flag:4734}}, init_mode_version:0, accumulate_checksum:2930602984, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326245982947004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17740, miss_cnt:0, cache_read_size:2170041, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:46.551465] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:50:46.551472] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58329][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:46.551795] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:50:46.553816] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCB-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:46.553845] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCB-0-0] [lt=23] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=32) [2025-01-08 00:50:46.555265] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1038048, replace_num=15728) [2025-01-08 00:50:46.555308] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:46.566087] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43AF-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:46.567118] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=44] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:50:46.578085] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=124] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:46.586855] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.605003] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=77, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:46.637771] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:46.644007] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=144] get_max_applied_scn(scn={val:1736326246399779001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18634142}, palf_committed_end_scn_:{val:1736326246399779002, v:0}, last_check_scn_:{val:1736326246399779002, v:0}, max_applied_cb_scn_:{val:1736326246399779001, v:0}}) [2025-01-08 00:50:46.644048] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18634142}, readable_log_tail:{lsn:18634142}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48470626, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:46.644290] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=216] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:46.666291] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B1-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:46.666415] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B1-0-0] [lt=47][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17829, log_task={header:{begin_lsn:{lsn:2170041}, end_lsn:{lsn:2170163}, log_id:17829, min_scn:{val:1736326246484620000, v:0}, max_scn:{val:1736326246484620000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2169919}, prev_proposal_id:1, committed_end_lsn:{lsn:2170041}, data_checksum:1715074492, accum_checksum:3448166050, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326246566051, freeze_ts:1736326246566051, submit_ts:1736326246566061, flushed_ts:1736326246569419, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:3358}, fs_cb_begin_ts=1736326246666345, log_life_time=100294) [2025-01-08 00:50:46.669520] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=114, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:46.669549] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:46.686536] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB29F-0-0] [lt=37] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246678461, keepalive_ts:-1}) [2025-01-08 00:50:46.686586] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB29F-0-0] [lt=48] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246678461, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:46.686992] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=110][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.687106] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A0-0-0] [lt=17] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246678550, keepalive_ts:-1}) [2025-01-08 00:50:46.687123] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A0-0-0] [lt=16] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246678550, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:46.697018] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50502][T1002_L0_G0][T1002][YB42C0A89F8A-00062B2D1BC9C774-0-0] [lt=4] get number(ret=0, service_type_=0, range=1, base_id=1736326246696225000, start_id=1736326246696225000, end_id=1736326246696225001) [2025-01-08 00:50:46.713137] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=22] ====== tenant freeze timer task ====== [2025-01-08 00:50:46.715260] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:46.743922] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=85] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326245982947004, v:0}, base_version={val:1736326245982947002, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326245982947004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326246743903, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326245982947004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326246766935, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326246193063002, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326246686030, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:50:46.755474] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:50:46.765569] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=40] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:46.773826] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=47] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:46.785137] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10183) [2025-01-08 00:50:46.787139] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [50576][T1001_RFLWorker][T1001][YB42C0A89F89-00062B2D1C78458E-0-0] [lt=109] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:50:46.787432] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.787675] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0ED-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246779340, keepalive_ts:-1}) [2025-01-08 00:50:46.787719] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0ED-0-0] [lt=42] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246779340, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:46.796619] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49CFF6-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:46.804322] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:46.808003] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=29] LogLoopThread round_cost_time(us)(round_cost_time=12) [2025-01-08 00:50:46.818469] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C7E-0-0] [lt=54][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:46.819718] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C7E-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:46.820544] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1736326246820541, bytes: 865262320, bw: 0.207148 MB/s, add_ts: 1009626, add_bytes: 219301 [2025-01-08 00:50:46.827650] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=43] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:46.827697] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=37] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=190) [2025-01-08 00:50:46.827736] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=110] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:46.827806] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=62] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:46.827821] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=184) [2025-01-08 00:50:46.834541] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2066) [2025-01-08 00:50:46.838382] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=224] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.844273] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4508) [2025-01-08 00:50:46.850671] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1002}, {id:1}, {id:1001}]) [2025-01-08 00:50:46.853266] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8534, offset=2420316}, end_cursor:ObLogCursor{file_id=1, log_id=8535, offset=2420621}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894648050, len:420, log_data_len:305, seq:8534, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:46.853621] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:46.853729] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=90] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326246696225001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326246696225001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:46.857623] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28B-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:46.857707] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28B-0-0] [lt=75] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326246484620002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326246484620002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:46.860588] INFO [LIB] stat (utility.h:1210) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=98, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:46.870719] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8535, offset=2420621}, end_cursor:ObLogCursor{file_id=1, log_id=8536, offset=2420918}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89464c050, len:4211, log_data_len:297, seq:8535, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:46.871452] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=21] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:46.871494] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326246696225001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326246696225001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:46.877971] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C28B-0-0] [lt=77] get number(ret=0, service_type_=0, range=1, base_id=1736326246874156000, start_id=1736326246874156001, end_id=1736326246874156002) [2025-01-08 00:50:46.883752] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58327][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B11-0-0] [lt=60] init point thread id with(&point=0x5572d50e1f00, idx_=983, point=[thread id=58327, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58327) [2025-01-08 00:50:46.884412] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8536, offset=2420918}, end_cursor:ObLogCursor{file_id=1, log_id=8537, offset=2421215}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894664050, len:3914, log_data_len:297, seq:8536, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:46.885872] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=71] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8537, offset=2421215}, end_cursor:ObLogCursor{file_id=1, log_id=8538, offset=2421512}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894650050, len:3617, log_data_len:297, seq:8537, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:46.886258] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17743, miss_cnt:0, cache_read_size:2170407, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:46.887541] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.892433] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-01-08 00:50:46.892461] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=104) [2025-01-08 00:50:46.897315] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.897583] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.897634] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.897711] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.900837] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:46.904891] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EBE-0-0] [lt=10] MediumCheckTask(cost_ts=9) [2025-01-08 00:50:46.916409] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=95] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:46.917665] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=45] ====== check clog disk timer task ====== [2025-01-08 00:50:46.917693] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:46.933447] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=2066, avg_group_log_size=206) [2025-01-08 00:50:46.933492] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2066) [2025-01-08 00:50:46.933513] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:46.939319] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2596, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:46.939381] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=230, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:46.939418] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=55] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:50:46.939437] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:46.939483] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:46.939579] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:46.944148] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=48] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:46.946886] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:46.947025] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:46.947984] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3679-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:46.950708] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58242][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=29] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:50:46.953642] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EE4-0-0] [lt=48] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=1765, avg_log_submit_to_slide_time=12466) [2025-01-08 00:50:46.953788] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=178] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.953829] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17657, miss_cnt:0, cache_read_size:2160082, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:46.955684] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:50:46.966819] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B7-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:46.966849] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B7-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22647, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:46.966860] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=2022, avg_log_submit_to_slide_time=22626) [2025-01-08 00:50:46.966989] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.976256] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326236961449, end_time=1736326246976254, dag_status={}) [2025-01-08 00:50:46.976287] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=27] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976300] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976308] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976316] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976323] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976330] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=33] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976419] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=54] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976426] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976433] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976445] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976453] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976461] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976470] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976478] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976485] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976493] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976501] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976509] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976517] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976525] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976533] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976541] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976549] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976557] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976564] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976572] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976580] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976588] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976596] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976604] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976611] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976619] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976627] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976635] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976642] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976650] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976658] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976666] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976674] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976682] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976689] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976697] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976705] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976712] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976728] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976787] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976796] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976804] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976812] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976820] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976828] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976836] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976844] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976852] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976859] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976867] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976875] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976882] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976890] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976898] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976906] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976913] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976921] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976929] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976936] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976944] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976952] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976960] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976967] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976975] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976983] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976991] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976998] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977006] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977014] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977022] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977030] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977037] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977045] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977053] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977062] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:50:46.977070] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:50:46.977077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:50:46.977083] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:50:46.977088] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:50:46.977093] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:50:46.977098] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:50:46.977103] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:50:46.977108] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:50:46.977113] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:50:46.977119] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:50:46.980466] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D2C-0-0] [lt=15] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:46.983750] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AB55-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.742487001419067383e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=387, Throttle Trigger(MB)=232, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.811310442503267950e+02) [2025-01-08 00:50:46.987526] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:46.987634] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44888, [delay/us]:prepare:8, schedule:63, submit:56, rt:1057, total:1185) [2025-01-08 00:50:46.987654] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:46.987728] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=58] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44893, [delay/us]: prepare:8, schedule:63, submit:56, device:1057, total:1185) [2025-01-08 00:50:46.987744] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:46.987783] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4742, iops:19, bw:89787, [delay/us]:prepare:7, schedule:137, submit:41, rt:1255, total:1441) [2025-01-08 00:50:46.987794] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:46.987831] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4743, avg_iops:19, avg_bw:89793, [delay/us]: prepare:7, schedule:137, submit:41, device:1255, total:1441) [2025-01-08 00:50:46.987842] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:46.987864] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20407, [delay/us]:prepare:7, schedule:91, submit:1, rt:1491, total:1591) [2025-01-08 00:50:46.987896] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:163262, [delay/us]:prepare:6, schedule:126, submit:56, rt:1317, total:1507) [2025-01-08 00:50:46.987908] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:46.987937] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20408, [delay/us]: prepare:7, schedule:91, submit:1, device:1491, total:1591) [2025-01-08 00:50:46.987953] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:163268, [delay/us]: prepare:6, schedule:126, submit:56, device:1317, total:1507) [2025-01-08 00:50:46.987991] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:46.988011] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4081, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:267, total:274) [2025-01-08 00:50:46.988056] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:46.988090] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4081, [delay/us]: prepare:0, schedule:0, submit:600000000, device:267, total:274) [2025-01-08 00:50:46.988109] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:46.988125] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:46.988133] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:46.988141] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:46.988148] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=282kB/s, net_out=237kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:46.988622] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.005588] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=4386, avg_group_log_size=438) [2025-01-08 00:50:47.005644] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4386) [2025-01-08 00:50:47.005662] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:47.007723] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B87-0-0] [lt=82] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:47.012183] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] Garbage Collector is running(seq_=192, gc_interval=10000000) [2025-01-08 00:50:47.012228] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=25] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:50:47.012243] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:50:47.015510] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:50:47.016617] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=24] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:50:47.016644] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=22] gc_check_member_list_ cost time(ret=0, time_us=4436) [2025-01-08 00:50:47.016660] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:50:47.018091] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:47.019466] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:47.020801] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:47.022087] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:47.022111] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:47.022127] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=15] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:47.022138] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:47.022149] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:47.022160] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] execute_gc cost time(ret=0, time_us=50) [2025-01-08 00:50:47.022169] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:50:47.022177] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-01-08 00:50:47.022258] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=70] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-01-08 00:50:47.045176] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x5572d527f900, last_check_time:1736326247045172, last_idle_time:14974162262, last_thread_count:4}, *pool={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:14974162262, tenant_id:500}, qsize=-4, current_idle_time=14974162262) [2025-01-08 00:50:47.045283] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=105] [thread pool stat](pool_stat={pool:0x2ad8885abcf0, last_check_time:1736326247045282, last_idle_time:5176894544, last_thread_count:2}, *pool={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5176894544, tenant_id:500}, qsize=-2, current_idle_time=5176894544) [2025-01-08 00:50:47.045297] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] [thread pool stat](pool_stat={pool:0x2ad8885acaf0, last_check_time:1736326247045297, last_idle_time:3743228072, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885acaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3743228072, tenant_id:500}, qsize=-1, current_idle_time=3743228072) [2025-01-08 00:50:47.045308] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8881fc3b0, last_check_time:1736326247045307, last_idle_time:14904819990, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:14904819990, tenant_id:1}, qsize=-4, current_idle_time=14904819990) [2025-01-08 00:50:47.045318] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8885ad1f0, last_check_time:1736326247045317, last_idle_time:3655319780, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885ad1f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3655319780, tenant_id:500}, qsize=-1, current_idle_time=3655319780) [2025-01-08 00:50:47.045327] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8c0eb0e70, last_check_time:1736326247045327, last_idle_time:7695491913, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7695491913, tenant_id:1002}, qsize=-4, current_idle_time=7695491913) [2025-01-08 00:50:47.045337] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8f90b0e70, last_check_time:1736326247045337, last_idle_time:7664844199, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7664844199, tenant_id:1001}, qsize=-4, current_idle_time=7664844199) [2025-01-08 00:50:47.045348] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5176894544, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:47.045361] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:47.045373] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad90857db90) [2025-01-08 00:50:47.045381] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5176894544, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:47.050762] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:47.051987] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:50:47.067228] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=51] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:47.067267] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=26] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=92) [2025-01-08 00:50:47.067983] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2174747, this={dio_aligned_buf:{buf_write_offset:3989, buf_padding_size:107, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:26062, truncate_used_ts:5364}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2170880, buf_len=122, total_write_size=11806, total_write_size_after_dio=405504, ob_pwrite_used_ts=613786, count=97, each_pwrite_cost=6327) [2025-01-08 00:50:47.075946] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46508][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18577D-0-0] [lt=41][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193751989779, cache_obj->added_lc()=false, cache_obj->get_object_id()=13544, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:47.076681] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46502][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18577E-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:47.078144] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=87][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:47.078277] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:47.078701] WDIAG get_refactored (ob_hashtable.h:1120) [46384][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185780-0-0] [lt=43][errcode=-4201] hashtable not init [2025-01-08 00:50:47.079255] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=24][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:47.088815] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.098845] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84146977}, readable_log_tail:{lsn:84146977}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:50062559, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:47.098942] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=95] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:47.099125] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=111] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:50:47.116998] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.118594] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=48] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=314570, clean_num=31457) [2025-01-08 00:50:47.129054] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=58][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:47.138169] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:47.139541] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=31] ====== checkpoint timer task ====== [2025-01-08 00:50:47.140506] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.140607] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=101] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.140720] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.140808] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=84] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:50:47.140828] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:47.140842] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326243478874001, v:0}) [2025-01-08 00:50:47.140850] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:50:47.140864] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:47.140883] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:47.140896] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:47.140915] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:47.140965] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=49] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:47.141822] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.141863] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=41] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.141907] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326241890747001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.141974] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=63] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:50:47.141988] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:50:47.142003] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:47.142018] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:47.142034] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2137625}, scn={val:1736326246900070001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:47.143132] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:316K/74:143741 disk:316K/74:143741 t1m2:44K/10:190471 t1m2g0:44K/10:190471 t1002m2:188K/46:136122 t1002m2g0:188K/46:136122 t1001m2:84K/18:137250 t1001m2g0:84K/18:137250 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:47.149268] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=61] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:47.149706] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB36-0-0] [lt=21][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193752063540, cache_obj->added_lc()=false, cache_obj->get_object_id()=13545, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:47.152800] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50899][T1002_L0_G2][T1002][YB42C0A89F8B-00062B2D1EABD762-0-0] [lt=30] get_min_unreplayed_log_info(lsn={lsn:2160204}, scn={val:1736326246693924004, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326247051172, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2160204}, next_to_submit_scn_:{val:1736326246693924004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2160082}, end_lsn:{lsn:2160204}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2160204}, readable_log_tail:{lsn:2160204}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64944564, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326246693924003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326246693924003, v:0}, data_checksum:2124447694, flag:19657}}, init_mode_version:0, accumulate_checksum:4202015717, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326246693924003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17658, miss_cnt:0, cache_read_size:2160204, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:47.154942] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Slow write(duration=12079, write_len=4096) [2025-01-08 00:50:47.154966] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=23] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8538, offset=2421512}, end_cursor:ObLogCursor{file_id=1, log_id=8539, offset=2421809}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89465c050, len:3320, log_data_len:297, seq:8538, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.155069] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] ====== check clog disk timer task ====== [2025-01-08 00:50:47.155094] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:47.155306] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:47.155389] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326246900070001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:47.155415] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:47.155444] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:50:47.156014] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=48] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:50:47.156196] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.156302] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=104] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.156398] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326241475765004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.156472] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=70] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:50:47.156487] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:50:47.156505] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:47.156520] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:47.156535] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2170773}, scn={val:1736326246984891000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:47.157317] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1864) [2025-01-08 00:50:47.159125] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=17] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8539, offset=2421809}, end_cursor:ObLogCursor{file_id=1, log_id=8540, offset=2422106}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894658050, len:3023, log_data_len:297, seq:8539, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.159277] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326246984891000, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:47.159382] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=98][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:47.159406] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:50:47.160281] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=59] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.160317] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=36] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.160360] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326241679125002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.160439] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=77] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:50:47.160454] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:50:47.160468] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:47.160482] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:47.160495] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2160204}, scn={val:1736326246693924003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:47.162360] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=2048) [2025-01-08 00:50:47.163690] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=47] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8540, offset=2422106}, end_cursor:ObLogCursor{file_id=1, log_id=8541, offset=2422403}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894654050, len:2726, log_data_len:297, seq:8540, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.163843] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326246693924003, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:47.163874] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:47.163894] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:50:47.163904] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:50:47.170419] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC35F-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.180569] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=50] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:47.181884] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=28] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:47.182031] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=42] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326244258401000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:47.188916] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.195918] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE7B5-0-0] [lt=279] get number(ret=0, service_type_=0, range=1, base_id=1736326247195038000, start_id=1736326247195038000, end_id=1736326247195038001) [2025-01-08 00:50:47.201131] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] ====== traversal_flush timer task ====== [2025-01-08 00:50:47.201301] INFO [MDS] flush (mds_table_mgr.cpp:195) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=41] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326246900070002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.201500] INFO [MDS] flush (mds_table_mgr.cpp:172) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=163] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246900070001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.201644] INFO [MDS] flush (mds_table_mgr.cpp:172) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=114] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246984891000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.201851] INFO [MDS] flush (mds_table_mgr.cpp:172) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=76] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246693924003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.201930] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=76] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:50:47.214367] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=39] gc stale ls task succ [2025-01-08 00:50:47.219738] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:47.219765] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:47.223346] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB86FFB-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.223391] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:47.223403] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:47.234481] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=95] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:47.235834] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:47.235947] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=41] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245675194000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:47.235975] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326245675194000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326245675194000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:47.243815] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=25] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:47.243838] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:47.245136] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848DF-0-0] [lt=8] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:47.245163] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848DF-0-0] [lt=24] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:47.248520] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC367F-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:47.248561] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC367F-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13064, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:47.249023] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:47.250029] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58330][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:47.250540] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:50:47.256531] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:47.257379] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=17] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:50:47.270817] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82A6-0-0] [lt=24] init point thread id with(&point=0x5572d50e1f80, idx_=985, point=[thread id=58329, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58329) [2025-01-08 00:50:47.272351] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C95-0-0] [lt=17] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:47.272398] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C95-0-0] [lt=43] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:47.272447] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=88] gc stale ls task succ [2025-01-08 00:50:47.272733] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E1-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:47.272787] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E1-0-0] [lt=37] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247195038000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326247195038000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:47.283456] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5E1-0-0] [lt=157] get number(ret=0, service_type_=0, range=1, base_id=1736326247282506000, start_id=1736326247282506000, end_id=1736326247282506001) [2025-01-08 00:50:47.285160] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540A-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.289055] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.293763] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42973173}], scn_array=[cnt:1, {val:1736326247282506000, v:0}], curr_size=750, accum_size=9596, time_cost=10012) [2025-01-08 00:50:47.293950] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=24] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:47.300767] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF24-0-0] [lt=67] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326247291830, keepalive_ts:-1}) [2025-01-08 00:50:47.300799] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF24-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326247291830, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:47.319224] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=70] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.319246] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.319284] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=63] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.327645] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=167] current monitor number(seq_=-1) [2025-01-08 00:50:47.333137] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:47.334433] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=47] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:47.334542] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=38] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245564663000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:47.335535] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:50:47.335695] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=156] try gc retain ctx [2025-01-08 00:50:47.335707] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] try to retry start_working [2025-01-08 00:50:47.335739] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326246900070003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326237108711001, v:0}, keep_alive_scn_:{val:1736326241890747004, v:0}, update_ts_:1736326247334299}) [2025-01-08 00:50:47.336561] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:47.336718] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=67, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:47.336764] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] finish update min start scn(max_decided_scn={val:1736326246900070003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326237108711000, v:0}, keep_alive_scn_:{val:1736326241890747004, v:0}, update_ts_:1736326247334299}) [2025-01-08 00:50:47.337489] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246900070001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:47.337633] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] finish update min start scn(max_decided_scn={val:1736326246984891001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326239968575000, v:0}, keep_alive_scn_:{val:1736326244981816000, v:0}, update_ts_:1736326247334299}) [2025-01-08 00:50:47.338557] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246984891000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:47.338626] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] finish update min start scn(max_decided_scn={val:1736326247195038000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326238175372002, v:0}, keep_alive_scn_:{val:1736326243184127002, v:0}, update_ts_:1736326247334299}) [2025-01-08 00:50:47.339333] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246693924003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:47.342339] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EEF-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:47.342375] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EEF-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4028, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:47.342893] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=81] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1634, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.349339] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3681-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:47.349373] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3681-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:47.351345] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=48] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=2324, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:47.351371] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10185) [2025-01-08 00:50:47.351389] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=379, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:47.351773] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=35, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:47.351831] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=28] gc stale ls task succ [2025-01-08 00:50:47.353051] INFO [LIB] stat (utility.h:1210) [58321][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37B1-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:47.353074] INFO [LIB] stat (utility.h:1210) [58321][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37B1-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4175, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:47.353084] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58321][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37B1-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=163, avg_log_gen_to_submit_time=270, avg_log_submit_to_flush_time=2912, avg_log_submit_to_slide_time=3905) [2025-01-08 00:50:47.353291] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=46] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326246693924003, v:0}, new_max_version={val:1736326246693924003, v:0}, last_min_version={val:1736326246484620003, v:0}, last_max_version={val:1736326246484620003, v:0}, total_time_us=5152, delta_us=659364) [2025-01-08 00:50:47.353321] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326246693924003, v:0}, min_version={val:1736326246693924003, v:0}, max_version={val:1736326246693924003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:47.353394] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10185, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.356186] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:50:47.367532] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43BF-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:47.374347] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=48] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:47.388672] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:47.388725] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=45] temporary file module use no blocks [2025-01-08 00:50:47.388738] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:47.388752] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:47.388815] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=61] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193751302361, flush_io_finished_ret_:0, flush_io_finished_round_:1906, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:47.389241] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.401660] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=86] get number(ret=0, service_type_=0, range=1, base_id=1736326247401274000, start_id=1736326247401274000, end_id=1736326247401274001) [2025-01-08 00:50:47.407483] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1519, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:47.407578] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=93] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3963) [2025-01-08 00:50:47.407603] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=407, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:47.407632] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=175] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:47.409855] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58330][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6701-0-0] [lt=26] init point thread id with(&point=0x5572d50e1fc0, idx_=986, point=[thread id=58330, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58330) [2025-01-08 00:50:47.409895] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A69-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:47.409919] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A69-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3391, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:47.409929] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A69-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=79, avg_log_submit_to_flush_time=2076, avg_log_submit_to_slide_time=3311) [2025-01-08 00:50:47.410004] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3963, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.411886] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=99] ====== traversal_flush timer task ====== [2025-01-08 00:50:47.412113] INFO [MDS] flush (mds_table_mgr.cpp:195) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EBF-0-0] [lt=42] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326247382286000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259284252000, latest_log_ts:{val:1736326242623339001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.412352] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EBF-0-0] [lt=194] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:50:47.417957] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=28] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:50:47.417992] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:50:47.418006] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:50:47.418017] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:47.420641] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=130, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}]) [2025-01-08 00:50:47.421036] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=390] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:47.429490] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-01-08 00:50:47.429522] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70285300, tenant_id=1001) [2025-01-08 00:50:47.429595] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=63] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:47.443956] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58265][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EF1-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=93, avg_log_submit_to_flush_time=2633, avg_log_submit_to_slide_time=4397) [2025-01-08 00:50:47.447937] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=91] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:47.448747] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:47.459557] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.459567] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=41] ====== tenant freeze timer task ====== [2025-01-08 00:50:47.459623] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.466021] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46493][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BEC0-0-0] [lt=53] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:50:47.468527] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1736326247468523, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007852, add_bytes: 0 [2025-01-08 00:50:47.471287] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2167, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:47.471339] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:47.471364] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=369, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:47.471436] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=41, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:47.489389] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.519308] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.519739] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.521683] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=101] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=10307, avg_group_log_size=542) [2025-01-08 00:50:47.521717] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10774) [2025-01-08 00:50:47.536258] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=204] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:47.536829] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=547] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:50:47.536852] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:50:47.536868] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:47.540095] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EF4-0-0] [lt=48] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:50:47.542382] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.542394] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2655, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:47.542446] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=537, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:47.542592] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:47.542616] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5464, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:47.544394] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.553434] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:47.553605] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:47.553847] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:47.556346] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:50:47.556612] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58265][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=30] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:50:47.567820] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43C3-0-0] [lt=69] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:47.578446] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:47.589519] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.600658] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B89-0-0] [lt=18] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:47.607659] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=55, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:47.638310] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=63] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:47.638624] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2142200, this={dio_aligned_buf:{buf_write_offset:114, buf_padding_size:3982, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:22333, truncate_used_ts:5026}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=8192, aligned_block_offset=2138112, buf_len=122, total_write_size=10342, total_write_size_after_dio=360448, ob_pwrite_used_ts=761632, count=85, each_pwrite_cost=8960) [2025-01-08 00:50:47.649741] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5315) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=26] is_in_sync(ret=0, palf_id=1003, is_in_sync=true, is_use_cache=true, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1736326247195038004, v:0}) [2025-01-08 00:50:47.651588] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58332][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:47.651678] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:50:47.651729] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1733, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:47.651757] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:47.651776] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=117, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:47.651827] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=54, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:47.668065] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43C5-0-0] [lt=46] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:47.670679] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=91, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:47.670707] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:47.671252] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=8] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:47.671304] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=36] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=91) [2025-01-08 00:50:47.678754] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=96] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):279, sequence:1}) [2025-01-08 00:50:47.683671] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=115] get_max_applied_scn(scn={val:1736326247585417000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:42976401}, palf_committed_end_scn_:{val:1736326247585417000, v:0}, last_check_scn_:{val:1736326247585417000, v:0}, max_applied_cb_scn_:{val:1736326247585417000, v:0}}) [2025-01-08 00:50:47.683724] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:42976401}, readable_log_tail:{lsn:42976401}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:24128367, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:47.689607] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.720358] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=86] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.756837] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:50:47.765769] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=41] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:47.774395] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=84] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:47.788984] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58321][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37B8-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10187) [2025-01-08 00:50:47.789754] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.796274] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49CFF9-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.804754] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=91] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:47.822734] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=117] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:50:47.828571] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1736326247828567, bytes: 865585587, bw: 0.305837 MB/s, add_ts: 1008026, add_bytes: 323267 [2025-01-08 00:50:47.829880] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C7F-0-0] [lt=56][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.832856] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C7F-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.834390] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A6D-0-0] [lt=15] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326247834373}) [2025-01-08 00:50:47.835346] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594D-0-0] [lt=23] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:47.840376] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594D-0-0] [lt=26] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:47.840421] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594D-0-0] [lt=42] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326247834373}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:47.840519] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:50:47.850445] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] ====== traversal_flush timer task ====== [2025-01-08 00:50:47.850849] INFO [MDS] flush (mds_table_mgr.cpp:195) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F8578F-0-0] [lt=32] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326247644191000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259254996000, latest_log_ts:{val:1736326242681587001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.850998] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F8578F-0-0] [lt=142] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:50:47.851037] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1003}, {id:1001}, {id:1002}]) [2025-01-08 00:50:47.851369] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.852805] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3539) [2025-01-08 00:50:47.858396] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1736326247844395000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1736326247858361}, version_delta=13999) [2025-01-08 00:50:47.859967] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=81] ls blacklist refresh finish(cost_time=27029) [2025-01-08 00:50:47.889945] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.890502] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=104] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8541, offset=2422403}, end_cursor:ObLogCursor{file_id=1, log_id=8542, offset=2422708}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894660050, len:2429, log_data_len:305, seq:8541, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.891024] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:47.891067] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247401274004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326247401274004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:47.896575] INFO [LIB] stat (utility.h:1210) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=54] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=100, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:47.896958] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.897261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.897435] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.897459] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.898378] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28C-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:47.898419] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28C-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247485899003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326247485899003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:47.902417] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=51] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326247902396], gts:1736326247901652000, latest_srr:[mts=1736326247902396]}) [2025-01-08 00:50:47.902449] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:47.902472] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326247902461], gts:1736326247901652001, latest_srr:[mts=1736326247902461]}) [2025-01-08 00:50:47.902714] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326247902702], gts:1736326247901652000, latest_srr:[mts=1736326247902702]}) [2025-01-08 00:50:47.906530] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=38] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8542, offset=2422708}, end_cursor:ObLogCursor{file_id=1, log_id=8543, offset=2423005}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894668050, len:2124, log_data_len:297, seq:8542, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.906892] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=82] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:47.906958] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=48] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247401274004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326247401274004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:47.921060] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=116] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.923537] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8543, offset=2423005}, end_cursor:ObLogCursor{file_id=1, log_id=8544, offset=2423302}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89466c050, len:1827, log_data_len:297, seq:8543, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.929825] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=50] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8544, offset=2423302}, end_cursor:ObLogCursor{file_id=1, log_id=8545, offset=2423599}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894674050, len:1530, log_data_len:297, seq:8544, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.930306] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17754, miss_cnt:0, cache_read_size:2171749, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:47.936714] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:50:47.936747] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:50:47.936764] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:47.939516] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:47.939986] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82B9-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:50:47.941237] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:50:47.941341] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:47.947872] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=63] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:47.948275] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=64] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:47.949143] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=12478464) [2025-01-08 00:50:47.950065] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:47.951444] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC368D-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:47.953308] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58321][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=48] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:50:47.957067] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=46] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:50:47.976607] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82BB-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=50, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:47.976657] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82BB-0-0] [lt=49] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12922, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:47.976625] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.976670] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82BB-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=40, avg_log_submit_to_flush_time=1852, avg_log_submit_to_slide_time=12882) [2025-01-08 00:50:47.984264] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB38-0-0] [lt=89][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193752898096, cache_obj->added_lc()=false, cache_obj->get_object_id()=13546, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:47.987143] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F451-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326247986924000, start_id=1736326247986924000, end_id=1736326247986924001) [2025-01-08 00:50:47.990290] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:47.990342] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44934, [delay/us]:prepare:26, schedule:572, submit:179, rt:1315, total:2093) [2025-01-08 00:50:47.990357] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:47.990474] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44935, [delay/us]: prepare:26, schedule:572, submit:179, device:1315, total:2093) [2025-01-08 00:50:47.990491] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:47.990585] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:17, bw:77606, [delay/us]:prepare:8, schedule:250, submit:236, rt:1573, total:2068) [2025-01-08 00:50:47.990600] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:47.990640] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:77606, [delay/us]: prepare:8, schedule:250, submit:236, device:1573, total:2068) [2025-01-08 00:50:47.990711] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=65] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:47.990738] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:7, bw:28589, [delay/us]:prepare:13, schedule:60, submit:1, rt:3562, total:3637) [2025-01-08 00:50:47.990761] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:171538, [delay/us]:prepare:7, schedule:180, submit:348, rt:1382, total:1919) [2025-01-08 00:50:47.990770] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12252, [delay/us]:prepare:1, schedule:129, submit:215, rt:218, total:565) [2025-01-08 00:50:47.990779] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:47.990810] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:7, avg_bw:28589, [delay/us]: prepare:13, schedule:60, submit:1, device:3562, total:3637) [2025-01-08 00:50:47.990826] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12252, [delay/us]: prepare:1, schedule:129, submit:215, device:218, total:565) [2025-01-08 00:50:47.990833] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:171537, [delay/us]: prepare:7, schedule:180, submit:348, device:1382, total:1919) [2025-01-08 00:50:47.990885] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:47.990909] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:251, total:259) [2025-01-08 00:50:47.990935] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:47.991008] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=67] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4084, [delay/us]: prepare:0, schedule:0, submit:600000000, device:251, total:259) [2025-01-08 00:50:47.991028] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:47.991043] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:47.991052] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:47.991059] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:47.991067] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=257kB/s, net_out=249kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:47.991176] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.994044] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB3B-0-0] [lt=40] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.791824254989624023e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=895, Throttle Trigger(MB)=537, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.762973993267839390e+01) [2025-01-08 00:50:47.996046] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB3D-0-0] [lt=65] get number(ret=0, service_type_=0, range=1, base_id=1736326247995951000, start_id=1736326247995951000, end_id=1736326247995951001) [2025-01-08 00:50:47.999876] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6707-0-0] [lt=27] init point thread id with(&point=0x5572d50e2040, idx_=988, point=[thread id=58332, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58332) [2025-01-08 00:50:48.000125] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=30] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:50:48.036308] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3B-0-0] [lt=14] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326248132022, keepalive_ts:-1}) [2025-01-08 00:50:48.036346] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3B-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326248132022, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:48.039287] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2026, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:48.039342] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=377, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:48.045410] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=40] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:48.045995] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53] no need to update min/max version(new_version={val:1736326247644191000, v:0}, min_version={val:1736326247644191000, v:0}, max_version={val:1736326247644191000, v:0}) [2025-01-08 00:50:48.050892] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC368F-0-0] [lt=15] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=2, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=2193, avg_log_submit_to_slide_time=31815) [2025-01-08 00:50:48.050998] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.051094] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17667, miss_cnt:0, cache_read_size:2161302, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:48.053670] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:48.054081] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:48.067737] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46100][TimerWK2][T0][YB42C0A89F89-00062B2D17E85432-0-0] [lt=69] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:50:48.078639] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=94] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:48.091297] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.101767] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326244089394, last_first_register_time_us=1736324449183670, register_time_us=1736326244089394, curr_time_us=1736326248101764) [2025-01-08 00:50:48.101898] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326248101805, last_submit_register_req_time_us=1736326248101805, register_parent_reason=1) [2025-01-08 00:50:48.101921] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:50:48.108451] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4104, avg_group_log_size=373) [2025-01-08 00:50:48.108481] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4104) [2025-01-08 00:50:48.108499] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:48.121429] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=278] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.138501] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:48.147828] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326248145826000, start_id=1736326248145826000, end_id=1736326248145826001) [2025-01-08 00:50:48.148292] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:324K/76:189932 disk:324K/76:189932 t1m2:48K/11:430733 t1m2g0:48K/11:430733 t1002m2:200K/48:117835 t1002m2g0:200K/48:117835 t1001m2:76K/17:237689 t1001m2g0:76K/17:237689 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:48.153745] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:48.157402] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:50:48.176793] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB3E-0-0] [lt=37][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193753090625, cache_obj->added_lc()=false, cache_obj->get_object_id()=13547, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:48.184322] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=107] no need to update min/max version(new_version={val:1736326247747226000, v:0}, min_version={val:1736326247747226000, v:0}, max_version={val:1736326247747226000, v:0}) [2025-01-08 00:50:48.186656] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC362-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.191432] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.201935] INFO [STORAGE] scheduler_transfer_handler_ (ob_transfer_service.cpp:202) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F0-0-0] [lt=21] start do transfer handler(ls_id_array_=[{id:1}, {id:1001}, {id:1002}, {id:1003}]) [2025-01-08 00:50:48.205461] WDIAG [SHARE] get_by_ls_id_ (ob_transfer_task_operator.cpp:754) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=29][errcode=-4018] dest ls transfer task not found(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1002, ls_id={id:1}) [2025-01-08 00:50:48.205587] WDIAG [SHARE] get_by_src_ls (ob_transfer_task_operator.cpp:189) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=21][errcode=-4018] failed to get by ls id(ret=-4018, tenant_id=1002, src_ls={id:1}) [2025-01-08 00:50:48.205605] WDIAG [STORAGE] fetch_transfer_task_from_inner_table_by_src_ls_ (ob_transfer_handler.cpp:246) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=18][errcode=-4018] failed to get transfer task(ret=-4018, tenant_id=1002, src_ls_id={id:1}) [2025-01-08 00:50:48.206821] WDIAG [SHARE] get_by_ls_id_ (ob_transfer_task_operator.cpp:754) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=8][errcode=-4018] dest ls transfer task not found(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1002, ls_id={id:1}) [2025-01-08 00:50:48.206896] WDIAG [SHARE] get_by_dest_ls (ob_transfer_task_operator.cpp:207) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=18][errcode=-4018] failed to get by ls id(ret=-4018, tenant_id=1002, dest_ls={id:1}) [2025-01-08 00:50:48.206913] WDIAG [STORAGE] fetch_transfer_task_from_inner_table_by_dest_ls_ (ob_transfer_handler.cpp:272) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=16][errcode=-4018] failed to get transfer task by dest ls(ret=-4018, tenant_id=1002, dest_ls_id={id:1}) [2025-01-08 00:50:48.209018] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:239) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=8] statistics the time that get needed to backfill tablets(ls id={id:1}, cost_time=2091) [2025-01-08 00:50:48.209039] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2386) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=16] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326245477976002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.210451] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:237) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=69] [TRANSFER] transfer handler process(ret=0, ls_id={id:1}) [2025-01-08 00:50:48.213594] WDIAG [SHARE] get_by_ls_id_ (ob_transfer_task_operator.cpp:754) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=17][errcode=-4018] dest ls transfer task not found(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1002, ls_id={id:1001}) [2025-01-08 00:50:48.213680] WDIAG [SHARE] get_by_src_ls (ob_transfer_task_operator.cpp:189) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=27][errcode=-4018] failed to get by ls id(ret=-4018, tenant_id=1002, src_ls={id:1001}) [2025-01-08 00:50:48.213696] WDIAG [STORAGE] fetch_transfer_task_from_inner_table_by_src_ls_ (ob_transfer_handler.cpp:246) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=15][errcode=-4018] failed to get transfer task(ret=-4018, tenant_id=1002, src_ls_id={id:1001}) [2025-01-08 00:50:48.214499] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:50:48.214765] WDIAG [SHARE] get_by_ls_id_ (ob_transfer_task_operator.cpp:754) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=8][errcode=-4018] dest ls transfer task not found(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1002, ls_id={id:1001}) [2025-01-08 00:50:48.214840] WDIAG [SHARE] get_by_dest_ls (ob_transfer_task_operator.cpp:207) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=18][errcode=-4018] failed to get by ls id(ret=-4018, tenant_id=1002, dest_ls={id:1001}) [2025-01-08 00:50:48.214911] WDIAG [STORAGE] fetch_transfer_task_from_inner_table_by_dest_ls_ (ob_transfer_handler.cpp:272) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=70][errcode=-4018] failed to get transfer task by dest ls(ret=-4018, tenant_id=1002, dest_ls_id={id:1001}) [2025-01-08 00:50:48.214984] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:239) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=13] statistics the time that get needed to backfill tablets(ls id={id:1001}, cost_time=57) [2025-01-08 00:50:48.214995] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2386) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=11] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246900070001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326245477976002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.216347] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:237) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=56] [TRANSFER] transfer handler process(ret=0, ls_id={id:1001}) [2025-01-08 00:50:48.217653] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:239) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F6-0-0] [lt=17] statistics the time that get needed to backfill tablets(ls id={id:1002}, cost_time=55) [2025-01-08 00:50:48.217673] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2386) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F6-0-0] [lt=18] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246984891000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326245477976002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.218938] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:237) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F6-0-0] [lt=56] [TRANSFER] transfer handler process(ret=0, ls_id={id:1002}) [2025-01-08 00:50:48.219920] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:48.219945] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:48.220255] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:239) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F8-0-0] [lt=17] statistics the time that get needed to backfill tablets(ls id={id:1003}, cost_time=115) [2025-01-08 00:50:48.220273] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2386) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F8-0-0] [lt=17] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246693924003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326245477976002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.221536] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:237) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F8-0-0] [lt=56] [TRANSFER] transfer handler process(ret=0, ls_id={id:1003}) [2025-01-08 00:50:48.224998] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87001-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.225066] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:48.225080] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:48.255073] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=42] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:50:48.255315] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=30] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:50:48.256771] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=51] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:48.264819] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=26] blacklist_loop exec finished(cost_time=21, is_enabled=true, send_cnt=0) [2025-01-08 00:50:48.274750] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C96-0-0] [lt=14] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:48.274790] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C96-0-0] [lt=37] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:48.274828] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:50:48.274920] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:48.274943] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:50:48.274951] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:48.274965] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:48.274979] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193752188332, flush_io_finished_ret_:0, flush_io_finished_round_:1911, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:48.287281] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540B-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.291599] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:48.291637] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.297327] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:50:48.297441] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:50:48.301995] INFO [CLOG] update_upstream (ob_remote_location_adaptor.cpp:91) [50440][T1002_LogRessvr][T1002][YB42C0A89F89-00062B2D1BA8458E-0-0] [lt=23] primary tenant, just skip [2025-01-08 00:50:48.303346] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=629149696, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:50:48.303377] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=29] === Run print tenant memory usage task === [2025-01-08 00:50:48.303424] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=13] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,612,800 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 631,246,848 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,248,302,215 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 160,801,160 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,038,594,048 max_mem_memstore_can_get_now= 804,005,886 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,248,302,215 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 70,285,300 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 656,101,376 max_mem_memstore_can_get_now= 351,426,548 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,248,302,215 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 241,092,180 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,110,196,224 max_mem_memstore_can_get_now= 1,205,460,968 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:50:48.303722] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=10] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,658,604,544 total_hold= 3,856,662,528 used= 3,467,284,480 freelists_hold= 191,320,064 total_maps= 1,624 total_unmaps= 232 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,612,800 virtual_memory_used= 4,829,216,768 [CHUNK_MGR] 2 MB_CACHE: hold= 88,080,384 count= 42 pushes= 1,803,412 pops= 1,803,370 maps= 1,493 unmaps= 227 [CHUNK_MGR] 4 MB_CACHE: hold= 9,035,776 count= 3 pushes= 6,152 pops= 6,149 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,058 pops= 3,054 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:50:48.303750] INFO print (ob_malloc_time_monitor.cpp:37) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=10] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 6144, delta_count= 2, avg_cost_time= 3072 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 375968, delta_count= 1, avg_cost_time= 375968 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-01-08 00:50:48.305952] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E2-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:48.306017] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E2-0-0] [lt=47] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247697287004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326247697287004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:48.319313] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.319357] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.322039] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=108] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.326067] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=20] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:48.337288] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB55-0-0] [lt=46] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:48.337345] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB55-0-0] [lt=49] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:48.337868] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=75, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:48.352017] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=15] gc stale ls task succ [2025-01-08 00:50:48.352975] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=45] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:48.353026] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=42] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:48.353057] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:48.353067] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=10] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29855, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:48.353114] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17759, log_task={header:{begin_lsn:{lsn:2161546}, end_lsn:{lsn:2161668}, log_id:17759, min_scn:{val:1736326248201055000, v:0}, max_scn:{val:1736326248201055000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2161424}, prev_proposal_id:1, committed_end_lsn:{lsn:2161546}, data_checksum:940009244, accum_checksum:4186338349, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326248250882, freeze_ts:1736326248250882, submit_ts:1736326248250892, flushed_ts:1736326248254901, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:4009}, fs_cb_begin_ts=1736326248353047, log_life_time=102165) [2025-01-08 00:50:48.353895] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=361] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2151, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:48.354120] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=222] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9051) [2025-01-08 00:50:48.354152] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=563, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:48.354365] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=61, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:48.357630] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=109] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:50:48.357658] INFO [LIB] stat (utility.h:1210) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37C1-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=35, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:48.357691] INFO [LIB] stat (utility.h:1210) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37C1-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4412, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:48.357702] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37C1-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=126, avg_log_submit_to_flush_time=2842, avg_log_submit_to_slide_time=4285) [2025-01-08 00:50:48.357775] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9051, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.358165] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326247697287003, v:0}, new_max_version={val:1736326247697287003, v:0}, last_min_version={val:1736326247401274004, v:0}, last_max_version={val:1736326247401274004, v:0}, total_time_us=7286, delta_us=660876) [2025-01-08 00:50:48.358309] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=139] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326247697287003, v:0}, min_version={val:1736326247697287003, v:0}, max_version={val:1736326247697287003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:48.368617] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43D3-0-0] [lt=180] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:48.389076] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=93] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:48.389115] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-01-08 00:50:48.389126] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:48.389139] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:48.389152] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193752302755, flush_io_finished_ret_:0, flush_io_finished_round_:1907, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:48.389545] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:48.389584] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-01-08 00:50:48.389592] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:48.389602] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:48.389613] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193752303070, flush_io_finished_ret_:0, flush_io_finished_round_:3735, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:48.391703] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.437573] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread began to shrink(tenant_->id()=1001, group_id_=20, token=0) [2025-01-08 00:50:48.442025] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F10-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:48.442062] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F10-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5706, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:48.442416] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.443306] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58034][T1001_L0_G20][T1001][Y0-0000000000000000-0-0] [lt=28] worker thread exit(tenant_->id()=1001, workers_.get_size()=1) [2025-01-08 00:50:48.446549] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:48.450367] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=131] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:48.450536] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=61] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:48.452372] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:42984152}, readable_log_tail:{lsn:42984152}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:24120616, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:48.452484] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=110] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:48.452518] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:50:48.453466] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=70] clear dtl interm result cost(us)(clear_cost=14, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:50:48.456281] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3697-0-0] [lt=128][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17760, log_task={header:{begin_lsn:{lsn:2161668}, end_lsn:{lsn:2161790}, log_id:17760, min_scn:{val:1736326248201055001, v:0}, max_scn:{val:1736326248201055001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2161546}, prev_proposal_id:1, committed_end_lsn:{lsn:2161668}, data_checksum:940009244, accum_checksum:1993249381, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326248352968, freeze_ts:1736326248352968, submit_ts:1736326248353019, flushed_ts:1736326248355569, gen_to_freeze cost time:0, gen_to_submit cost time:51, submit_to_flush cost time:2550}, fs_cb_begin_ts=1736326248456269, log_life_time=103301) [2025-01-08 00:50:48.458200] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.459352] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEBD-0-0] [lt=31] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:48.459391] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEBD-0-0] [lt=31] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:48.459571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.468315] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=27] clear dtl interm result cost(us)(clear_cost=3678, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:50:48.469277] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43D5-0-0] [lt=131][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17847, log_task={header:{begin_lsn:{lsn:2172237}, end_lsn:{lsn:2172359}, log_id:17847, min_scn:{val:1736326247986924003, v:0}, max_scn:{val:1736326247986924003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2172115}, prev_proposal_id:1, committed_end_lsn:{lsn:2172237}, data_checksum:1715074492, accum_checksum:3601824095, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326248368601, freeze_ts:1736326248368601, submit_ts:1736326248368749, flushed_ts:1736326248370041, gen_to_freeze cost time:0, gen_to_submit cost time:148, submit_to_flush cost time:1292}, fs_cb_begin_ts=1736326248469264, log_life_time=100663) [2025-01-08 00:50:48.469354] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58333][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:48.469545] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:50:48.475903] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=90] PNIO [ratelimit] time: 1736326248475900, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007377, add_bytes: 0 [2025-01-08 00:50:48.491784] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.514157] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2240, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:48.514245] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=87] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4104) [2025-01-08 00:50:48.514310] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=164, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:48.514711] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:48.514752] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4214, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:48.514766] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=76, avg_log_submit_to_flush_time=2603, avg_log_submit_to_slide_time=4138) [2025-01-08 00:50:48.514800] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=30, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:48.514862] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4104, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.519441] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=85] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.522502] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.526430] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9051, avg_group_log_size=565) [2025-01-08 00:50:48.526458] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9051) [2025-01-08 00:50:48.539244] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:48.539296] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:48.539317] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:48.539333] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:48.541768] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:48.542529] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.543917] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F12-0-0] [lt=84] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:48.543944] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F12-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5096, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:48.544404] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.544505] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F13-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=2790, avg_log_submit_to_slide_time=5375) [2025-01-08 00:50:48.552618] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3699-0-0] [lt=156] init point thread id with(&point=0x5572d50e2080, idx_=989, point=[thread id=58333, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58333) [2025-01-08 00:50:48.553921] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:48.554463] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=80] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:48.557877] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:48.559415] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCC-0-0] [lt=23] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:48.559447] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCC-0-0] [lt=24] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=34) [2025-01-08 00:50:48.570353] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2073, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:48.570417] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=62] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:48.570446] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=150, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:48.570520] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=190] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=59, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:48.572472] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D5E2-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:50:48.579022] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:48.580008] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:62, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2012642216, recv_hp_rpc_cnt:5121, recv_np_rpc_cnt:102763, recv_lp_rpc_cnt:0, recv_mysql_cnt:9246, recv_task_cnt:3700, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10715977, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=115021 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1823, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:93030, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736326247650243}{group_id:10, queue_size:0, recv_req_cnt:11085, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2057, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326228853642}{group_id:2, queue_size:0, recv_req_cnt:29786, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11811, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:4932, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:26471, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=308 pcode=0x1502:cnt=210 pcode=0x717:cnt=189 pcode=0x710:cnt=67 pcode=0x150b:cnt=60, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:50:48.580949] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=937] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:13917211, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8129, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3545766, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=23, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:50:48.581634] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=685] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:13089351, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3532597, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:50:48.582259] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=623] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:23, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:863494491, recv_hp_rpc_cnt:3668, recv_np_rpc_cnt:37254, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:764502, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=64531 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:66684, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:6, nesting_worker_cnt:0, token_change:1736326247950061}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2144, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326218958545}{group_id:2, queue_size:0, recv_req_cnt:15181, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18209, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=350 pcode=0x51c:cnt=247 pcode=0x717:cnt=183 pcode=0x710:cnt=98 pcode=0x150b:cnt=60, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:50:48.582958] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=697] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:60, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9225092389, recv_hp_rpc_cnt:77, recv_np_rpc_cnt:34281, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2891189, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=24661 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:0, recv_req_cnt:6887, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326208268040}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:147538, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:8, nesting_worker_cnt:0, token_change:1736326248468954}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326242277705}{group_id:19, queue_size:0, recv_req_cnt:396, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326204280596}{group_id:2, queue_size:0, recv_req_cnt:36376, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:10880, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=436 pcode=0x51c:cnt=400 pcode=0x1501:cnt=199 pcode=0x717:cnt=192 pcode=0x1516:cnt=186, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:50:48.591963] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.597857] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] BlockGCTimerTask success(ret=0, cost_time_us=10, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):104, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:50:48.604676] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=852][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:50:48.638712] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=69] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:48.639995] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=30] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:48.640828] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2493, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:48.640961] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=946, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:48.643614] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=36] [APPLY STAT CB APPEND COST TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=148, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:50:48.643640] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=26] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=4666, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:50:48.643649] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:50:48.643657] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=-1226, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:50:48.643664] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=3588, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:50:48.655699] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:50:48.665372] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58330][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=35] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:50:48.669657] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43D9-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:48.670431] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326238656814, end_time=1736326248670429, dag_status={}) [2025-01-08 00:50:48.670534] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=53] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670630] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=95] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670639] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670647] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670654] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670661] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670668] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670675] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670681] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670689] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670701] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670709] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670717] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670725] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670733] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670741] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670749] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670757] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670765] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670773] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670781] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670789] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670797] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670845] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=47] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670856] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670864] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670871] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670880] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670888] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670896] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670904] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670912] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670920] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670928] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670936] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670943] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671050] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=105] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671060] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671068] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671076] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671084] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671092] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671100] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671108] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671116] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671124] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671132] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671140] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671148] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671156] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671165] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671173] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671257] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671269] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671278] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671285] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671293] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671301] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671309] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671317] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671333] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671340] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671348] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671356] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671364] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671372] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671380] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671388] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671395] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671403] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671411] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671419] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671427] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671434] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671442] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671451] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671459] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671466] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671474] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671482] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671490] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671500] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:50:48.671508] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:50:48.671516] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:50:48.671521] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:50:48.671526] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:50:48.671532] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:50:48.671537] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:50:48.671542] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:50:48.671547] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:50:48.671552] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:50:48.671558] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:50:48.681973] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDB7-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193753595804, cache_obj->added_lc()=false, cache_obj->get_object_id()=13548, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:48.685246] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=91] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326248386431000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=298803, in_cluster_service=true, cluster_version={val:1736326248350690000, v:0}, min_cluster_version={val:1736326248350690000, v:0}, max_cluster_version={val:1736326248350690000, v:0}, get_cluster_version_err=0, cluster_version_delta=334544, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18494, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=594322017, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:48.692097] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.698899] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=32] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:50:48.700827] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1736326248699757000, start_id=1736326248699757000, end_id=1736326248699757001) [2025-01-08 00:50:48.700991] INFO [LIB] stat (utility.h:1210) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=59, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:48.722870] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.726597] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:50:48.726645] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=45] try gc retain ctx [2025-01-08 00:50:48.726709] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=63] try to retry start_working [2025-01-08 00:50:48.726746] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326248586728000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326238498963001, v:0}, keep_alive_scn_:{val:1736326243676959001, v:0}, update_ts_:1736326248726426}) [2025-01-08 00:50:48.727913] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=93] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259284252000, latest_log_ts:{val:1736326242623339001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.740414] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=18] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:48.740469] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:44332, max_lsn:84153042, max_scn:{val:1736326248699757000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84153042}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42210002}}, last_submit_lsn:{lsn:84152497}, last_submit_end_lsn:{lsn:84153042}, last_submit_log_id:44332, last_submit_log_pid:1, max_flushed_lsn:{lsn:84152497}, max_flushed_end_lsn:{lsn:84153042}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84153042}, last_slide_log_id:44332, last_slide_scn:{val:1736326248699757000, v:0}, last_slide_lsn:{lsn:84152497}, last_slide_end_lsn:{lsn:84153042}, last_slide_log_pid:1, last_slide_log_accum_checksum:195906776, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:44333, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84153042}, readable_log_tail:{lsn:84153042}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:50056494, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:50:48.741025] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=550] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248706534, last_flushed_end_lsn:{lsn:84153042}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248704977, last_flushed_end_lsn:{lsn:84153042}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248704866, last_flushed_end_lsn:{lsn:84153042}}]) [2025-01-08 00:50:48.741051] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=44333, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248706534, last_flushed_end_lsn:{lsn:84153042}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248704977, last_flushed_end_lsn:{lsn:84153042}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248704866, last_flushed_end_lsn:{lsn:84153042}}]) [2025-01-08 00:50:48.741421] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=26] ====== tenant freeze timer task ====== [2025-01-08 00:50:48.742459] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42985452}], scn_array=[cnt:1, {val:1736326248586728001, v:0}], curr_size=113, accum_size=2002, time_cost=14146) [2025-01-08 00:50:48.743853] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=93] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:50:48.745672] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=82] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:48.751447] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326248201055003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=550382, in_cluster_service=true, cluster_version={val:1736326247901652003, v:0}, min_cluster_version={val:1736326247901652003, v:0}, max_cluster_version={val:1736326247901652003, v:0}, get_cluster_version_err=0, cluster_version_delta=849785, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18058, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=476668964, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:48.755923] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2169, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:48.755966] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1333) [2025-01-08 00:50:48.755987] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=378, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:48.756040] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=64, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:48.759120] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=7] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1053776, replace_num=15728) [2025-01-08 00:50:48.759149] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:48.761805] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708513F-0-0] [lt=38] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:48.765958] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=45] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:48.769102] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=33] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=578, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:48.769129] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:48.774607] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=38] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:48.791526] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58207][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B30-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9042) [2025-01-08 00:50:48.794665] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.800960] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49CFFC-0-0] [lt=27][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.803880] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] LogLoopThread round_cost_time(us)(round_cost_time=27) [2025-01-08 00:50:48.804058] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=80][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42986154}], scn_array=[cnt:1, {val:1736326248785299000, v:0}], curr_size=588, accum_size=3179, time_cost=16141) [2025-01-08 00:50:48.805009] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:48.812415] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:50:48.812465] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=48] try gc retain ctx [2025-01-08 00:50:48.812476] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-01-08 00:50:48.812504] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1736326248747804000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84153628}, palf_committed_end_scn_:{val:1736326248747804000, v:0}, last_check_scn_:{val:1736326248747804000, v:0}, max_applied_cb_scn_:{val:1736326248747804000, v:0}}) [2025-01-08 00:50:48.812528] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84153628}, readable_log_tail:{lsn:84153628}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:50055908, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:48.812636] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=104] finish update min start scn(max_decided_scn={val:1736326248747804000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326238679896001, v:0}, keep_alive_scn_:{val:1736326243686095001, v:0}, update_ts_:1736326248811270}) [2025-01-08 00:50:48.813704] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259254996000, latest_log_ts:{val:1736326242681587001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.830414] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1736326248830410, bytes: 865840509, bw: 0.242665 MB/s, add_ts: 1001843, add_bytes: 254922 [2025-01-08 00:50:48.841702] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C80-0-0] [lt=68][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.842503] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=268] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:48.843627] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30] LogLoopThread round_cost_time(us)(round_cost_time=634) [2025-01-08 00:50:48.845993] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C80-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.848362] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5178613819, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:48.848396] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:48.848408] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad9085995f0) [2025-01-08 00:50:48.848417] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5178613819, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:48.851270] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}, {id:1003}]) [2025-01-08 00:50:48.892671] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=95] ====== check clog disk timer task ====== [2025-01-08 00:50:48.892788] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=65] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=104) [2025-01-08 00:50:48.894764] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=91][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.897274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=75] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.897365] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.897411] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.897478] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.904040] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=75] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:48.904382] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=502, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:48.911649] INFO [STORAGE.COMPACTION] schedule_all_tablets_medium (ob_tenant_tablet_scheduler.cpp:1597) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=17] start schedule all tablet merge(merge_version=1) [2025-01-08 00:50:48.912132] INFO [STORAGE] get_adaptive_merge_reason (ob_partition_merge_policy.cpp:1719) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=95] Check tablet adaptive merge reason(ret=0, ls_id={id:1}, tablet_id={id:60476}, reason=0, tablet_analyzer={tablet_stat:{ls_id:1, tablet_id:60476, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, total_tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_small_tenant:false, boost_factor:"1.000000000000000000e+00"}, crazy_medium_flag=false) [2025-01-08 00:50:48.913853] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=25] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:50:48.915644] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4519) [2025-01-08 00:50:48.923307] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=61] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.925940] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=35] ====== check clog disk timer task ====== [2025-01-08 00:50:48.925982] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=37] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:48.927607] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=166] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=76, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:48.935228] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=42] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8545, offset=2423599}, end_cursor:ObLogCursor{file_id=1, log_id=8546, offset=2423904}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894670050, len:1233, log_data_len:305, seq:8545, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:48.936101] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:48.936297] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=42] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326248403491004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326248403491004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:48.940595] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:48.940626] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:48.940642] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:48.941364] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:491) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=8] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=0, MAX_GAP=900000000000, new_sync_scn={val:1736326247401274004, v:0}, new_replayable_scn={val:1736326247401274004, v:0}, new_readable_scn={val:1736326247401274004, v:0}, old_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326246484620002, replayable_scn:1736326246484620002, readable_scn:1736326246484620002, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}) [2025-01-08 00:50:48.941803] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326248940813000, start_id=1736326248940813000, end_id=1736326248940813001) [2025-01-08 00:50:48.942692] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28D-0-0] [lt=59] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:48.942730] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28D-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326248488325003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326248488325003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:48.943083] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=22] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:48.945100] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=186] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:48.945563] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:48.945833] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=223] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:48.946072] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82D8-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:50:48.947035] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.950244] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=73] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8546, offset=2423904}, end_cursor:ObLogCursor{file_id=1, log_id=8547, offset=2424201}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894678050, len:928, log_data_len:297, seq:8546, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:48.951255] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:48.951386] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:48.951607] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:48.952646] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58275][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=108] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:50:48.955287] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36A1-0-0] [lt=47] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:50:48.956200] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=68] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:48.956250] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326248403491004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326248403491004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:48.960060] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:50:48.964450] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18637240}], scn_array=[cnt:1, {val:1736326248903338000, v:0}], curr_size=122, accum_size=122, time_cost=21916) [2025-01-08 00:50:48.965237] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:48.970699] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8547, offset=2424201}, end_cursor:ObLogCursor{file_id=1, log_id=8548, offset=2424498}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89467c050, len:631, log_data_len:297, seq:8547, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:48.971947] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C28D-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326248971489000, start_id=1736326248971489000, end_id=1736326248971489001) [2025-01-08 00:50:48.972390] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17765, miss_cnt:0, cache_read_size:2173091, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:48.974818] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8548, offset=2424498}, end_cursor:ObLogCursor{file_id=1, log_id=8549, offset=2424795}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894680050, len:4430, log_data_len:297, seq:8548, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:48.982857] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB40-0-0] [lt=109][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193753896639, cache_obj->added_lc()=false, cache_obj->get_object_id()=13549, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:48.987293] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D2D-0-0] [lt=94] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:48.988984] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42989337}], scn_array=[cnt:1, {val:1736326248971489000, v:0}], curr_size=750, accum_size=6524, time_cost=16602) [2025-01-08 00:50:48.990727] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58213][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB5B-0-0] [lt=26] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:48.990771] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58213][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB5B-0-0] [lt=35] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:48.995147] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:48.995272] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=121][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.995341] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:48.995399] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:55, bw:224145, [delay/us]:prepare:1, schedule:540, submit:0, rt:530, total:1073) [2025-01-08 00:50:48.995435] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:52979, [delay/us]:prepare:10, schedule:195, submit:72, rt:1768, total:2047) [2025-01-08 00:50:48.995454] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=3, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=13, ops=3, iops=16, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:48.995540] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:55, avg_bw:224129, [delay/us]: prepare:1, schedule:540, submit:0, device:530, total:1073) [2025-01-08 00:50:48.995565] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52975, [delay/us]: prepare:10, schedule:195, submit:72, device:1768, total:2047) [2025-01-08 00:50:48.995591] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:48.995635] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:17, bw:77432, [delay/us]:prepare:7, schedule:185, submit:1068, rt:3045, total:4307) [2025-01-08 00:50:48.995657] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:48.995827] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=148] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:77422, [delay/us]: prepare:7, schedule:185, submit:1068, device:3045, total:4307) [2025-01-08 00:50:48.995854] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:48.995900] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20374, [delay/us]:prepare:4, schedule:53, submit:0, rt:2511, total:2569) [2025-01-08 00:50:48.995937] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4198, iops:40, bw:167074, [delay/us]:prepare:7, schedule:142, submit:656, rt:1998, total:2805) [2025-01-08 00:50:48.995958] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:48.996122] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=154] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20371, [delay/us]: prepare:4, schedule:53, submit:0, device:2511, total:2569) [2025-01-08 00:50:48.996146] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4198, avg_iops:40, avg_bw:167048, [delay/us]: prepare:7, schedule:142, submit:656, device:1998, total:2805) [2025-01-08 00:50:48.996157] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:48.996179] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4074, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:303, total:312) [2025-01-08 00:50:48.996259] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:48.996297] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4074, [delay/us]: prepare:0, schedule:0, submit:600000000, device:303, total:312) [2025-01-08 00:50:48.996317] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:48.996333] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:48.996392] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=56] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:48.996401] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:48.996409] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=239kB/s, net_out=218kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:48.999990] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=76] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:50:49.001714] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326249001380000, start_id=1736326249001380000, end_id=1736326249001380001) [2025-01-08 00:50:49.008403] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=30] ====== checkpoint timer task ====== [2025-01-08 00:50:49.009276] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=45] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:49.009395] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=117] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:49.009572] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=49] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259284252000, latest_log_ts:{val:1736326242623339001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:49.009838] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=260] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:50:49.009925] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=79] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:49.009957] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] get rec log scn(service_type_=0, rec_log_ts={val:1736326242623339001, v:0}) [2025-01-08 00:50:49.009969] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:50:49.009976] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:50:49.009988] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:49.010007] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:49.010063] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=52] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:49.010085] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:49.010107] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:49.010135] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=26] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:50:49.038145] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3C-0-0] [lt=128] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326249134834, keepalive_ts:-1}) [2025-01-08 00:50:49.038313] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3C-0-0] [lt=164] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326249134834, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:49.043315] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4597, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:49.043349] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=816, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:49.051476] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36A3-0-0] [lt=27] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=2390, avg_log_submit_to_slide_time=42664) [2025-01-08 00:50:49.051746] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.051785] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17677, miss_cnt:0, cache_read_size:2162513, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:49.054815] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=44] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:49.062864] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:49.071570] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82DC-0-0] [lt=31] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=23, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:49.071613] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82DC-0-0] [lt=42] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38181, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:49.071625] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82DC-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=25, avg_log_submit_to_flush_time=2560, avg_log_submit_to_slide_time=38156) [2025-01-08 00:50:49.072120] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.079281] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=49] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:49.088663] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=18] success to update min reserved snapshot(reserved_snapshot=1736324444258401000, duration=1800, snapshot_gc_ts=1736326244258401000) [2025-01-08 00:50:49.093480] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=54] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324444258401000) [2025-01-08 00:50:49.093533] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=50] update reserved snapshot finished(cost_ts=4875, reserved_snapshot=1736324444258401000) [2025-01-08 00:50:49.095300] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.105295] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=36] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326245095005, last_first_register_time_us=1736324454612308, register_time_us=1736326245095005, curr_time_us=1736326249105291) [2025-01-08 00:50:49.105362] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=36] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326249105334, last_submit_register_req_time_us=1736326249105334, register_parent_reason=1) [2025-01-08 00:50:49.105380] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:50:49.114515] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=99] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4076, avg_group_log_size=370) [2025-01-08 00:50:49.114547] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4076) [2025-01-08 00:50:49.114608] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:49.119520] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58213][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC3-0-0] [lt=29] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:49.119581] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58213][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC3-0-0] [lt=52] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:49.124685] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.126356] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=51] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=346027, clean_num=31457) [2025-01-08 00:50:49.145596] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=49] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:49.149895] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690D1B-0-0] [lt=33] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.794224309921264648e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=897, Throttle Trigger(MB)=538, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.748260800254538339e+01) [2025-01-08 00:50:49.150764] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=27] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:1408K/347:176556 disk:1408K/347:176556 t1m2:1140K/284:172755 t1m2g0:1140K/284:172755 t1002m2:184K/44:155021 t1002m2g0:184K/44:155021 t1001m2:84K/19:175093 t1001m2g0:84K/19:175093 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:49.154279] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=57] get number(ret=0, service_type_=0, range=1, base_id=1736326249153254000, start_id=1736326249153254000, end_id=1736326249153254001) [2025-01-08 00:50:49.154450] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:49.155345] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-01-08 00:50:49.155373] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:49.160340] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:50:49.162508] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=30] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:50:49.164373] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326249162225000, start_id=1736326249162225000, end_id=1736326249162225001) [2025-01-08 00:50:49.170388] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18641702, this={dio_aligned_buf:{buf_write_offset:1351, buf_padding_size:2745, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:32013, truncate_used_ts:6657}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=18640896, buf_len=545, total_write_size=185050, total_write_size_after_dio=696320, ob_pwrite_used_ts=358235, count=125, each_pwrite_cost=2865) [2025-01-08 00:50:49.175430] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC365-0-0] [lt=28][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.195395] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.195436] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB42-0-0] [lt=21][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193754109267, cache_obj->added_lc()=false, cache_obj->get_object_id()=13550, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:49.214660] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:50:49.220127] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:49.220181] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:49.223349] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87007-0-0] [lt=14][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.223395] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:49.223406] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:49.246090] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=45] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:49.246116] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:49.246239] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E0-0-0] [lt=99] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:49.246282] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E0-0-0] [lt=40] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:49.258335] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:49.277740] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C97-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:49.277778] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C97-0-0] [lt=35] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:49.277815] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:50:49.277860] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:49.277880] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-01-08 00:50:49.277888] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:49.277901] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:49.277915] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193753188851, flush_io_finished_ret_:0, flush_io_finished_round_:1912, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:49.289382] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540C-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.295426] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.295957] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=8] finish to resize log external storage handler(current_ts=1736326249295952, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:50:49.295992] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=32] snapshot_traffic_info once [2025-01-08 00:50:49.308349] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBC-0-0] [lt=86] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326249405156, keepalive_ts:-1}) [2025-01-08 00:50:49.308420] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBC-0-0] [lt=70] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326249405156, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:49.311116] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=29] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:50:49.311163] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=41] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:50:49.311361] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=194] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:50:49.311376] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:49.314079] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:50:49.315120] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=1036] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:49.319323] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.319365] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.327130] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.329435] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-01-08 00:50:49.336160] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E3-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:49.336303] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E3-0-0] [lt=124] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326249201328000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326249201328000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:49.341178] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=73, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:49.342706] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=33] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:50:49.353162] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36A9-0-0] [lt=26] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:49.353583] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=28] gc stale ls task succ [2025-01-08 00:50:49.356249] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=4706, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:49.356395] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=144] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9781) [2025-01-08 00:50:49.356430] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=239, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:49.356506] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=29, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:49.357475] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58207][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B3D-0-0] [lt=17] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=20, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=5095, avg_log_submit_to_slide_time=5880) [2025-01-08 00:50:49.357600] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9781, hit_cnt=21, read_cnt=21, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.358646] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326248700933003, v:0}, new_max_version={val:1736326248700933003, v:0}, last_min_version={val:1736326248488325003, v:0}, last_max_version={val:1736326248488325003, v:0}, total_time_us=6200, delta_us=657711) [2025-01-08 00:50:49.358679] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326248700933003, v:0}, min_version={val:1736326248700933003, v:0}, max_version={val:1736326248700933003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:49.360631] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:50:49.361042] INFO [LIB] stat (utility.h:1210) [58307][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37D6-0-0] [lt=84] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:49.361064] INFO [LIB] stat (utility.h:1210) [58307][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37D6-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=5884, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:49.361471] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=24] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:49.365920] INFO [STORAGE] find_adaptive_merge_tables (ob_partition_merge_policy.cpp:1601) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=40] no enough table or no enough rows for meta merge(ret=-4677, scanty_tx_determ_table=false, scanty_inc_row_cnt=true, inc_row_cnt=9, base_inc_row_cnt=0, result={version_range:{multi_version_start:1736322630524987000, base_version:0, snapshot_version:1736322630524987000}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1736322632627436002, v:0}}, merge_version:0, is_simplified:false, handle:{tablet_id:{id:200001}, table_count:2, [{i:0, table_key:{tablet_id:{id:200001}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1, v:0}}}, ref:0}{i:1, table_key:{tablet_id:{id:200001}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1736322632627436002, v:0}}}, ref:0}]}, update_tablet_directly:false, schedule_major:false, is_backfill:false, backfill_scn:{val:18446744073709551615, v:3}, transfer_seq:-1}, PRINT_TS_WRAPPER(table_store_wrapper)={ObTabletTableStore_Pretty:this:0x2ad8a9350838, major_tables:{cnt:1, sstable_array:0x2ad8ebbea890, serialize_table_type:false, is_inited:true}, minor_tables:{cnt:1, sstable_array:0x2ad8ebbeac60, serialize_table_type:false, is_inited:true}, mds_sstables:{cnt:1, sstable_array:0x2ad8ebbeb686, serialize_table_type:false, is_inited:true}, memtables:{count:0}, is_ready_for_read:true, table_array:{ [46307][T1_TimerWK3][T1] [YB42C0A89F89-00062B2D16E8574F-0-0] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [46307][T1_TimerWK3][T1] [YB42C0A89F89-00062B2D16E8574F-0-0] MAJOR MAJOR 0 1 0 1 0 false [46307][T1_TimerWK3][T1] [YB42C0A89F89-00062B2D16E8574F-0-0] MINOR MINI 1736322630524987000 1736322630524987000 1 1736322632627436002 0 false [46307][T1_TimerWK3][T1] [YB42C0A89F89-00062B2D16E8574F-0-0] MDS MDS_MINI 1736322603189421000 1736322603189421000 1 1736322605489896000 0 false }}) [2025-01-08 00:50:49.372522] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] succ to submit log task to replay service(to_submit_lsn={lsn:2173513}, to_submit_scn={val:1736326248991181003, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326249372377, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2173579}, next_to_submit_scn_:{val:1736326248991181004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2173457}, end_lsn:{lsn:2173579}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2173579}, readable_log_tail:{lsn:2173579}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64931189, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326248991181003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326248991181003, v:0}, data_checksum:3348837161, flag:27546}}, init_mode_version:0, accumulate_checksum:988638107, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326248991181003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17769, miss_cnt:0, cache_read_size:2173579, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:49.389483] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=141] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:49.389524] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-01-08 00:50:49.389534] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:49.389592] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=54] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:49.389606] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193753303140, flush_io_finished_ret_:0, flush_io_finished_round_:1908, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:49.392945] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=95] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:49.393109] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=158] temporary file module use no blocks [2025-01-08 00:50:49.393119] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:49.393131] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:49.393143] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193753303554, flush_io_finished_ret_:0, flush_io_finished_round_:3736, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:49.395514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.404925] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=80] ts response task statistics(alloc_count=59382, free_count=29690) [2025-01-08 00:50:49.407092] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=23] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:49.418574] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=41] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:49.435565] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] success to update min reserved snapshot(reserved_snapshot=1736324445564663000, duration=1800, snapshot_gc_ts=1736326245564663000) [2025-01-08 00:50:49.438714] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=44] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324445564663000) [2025-01-08 00:50:49.439708] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12] ====== tenant freeze timer task ====== [2025-01-08 00:50:49.439741] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70285300, tenant_id=1001) [2025-01-08 00:50:49.439763] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:49.443735] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27] submit reserved snapshot log success(ls_id={id:1001}, new_snapshot_version=1736324445564663000) [2025-01-08 00:50:49.443768] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=29] update reserved snapshot finished(cost_ts=8208, reserved_snapshot=1736324445564663000) [2025-01-08 00:50:49.446122] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F2E-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:49.446160] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F2E-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6769, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:49.446453] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=126] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1755, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.451545] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=302] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:49.452878] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:49.453533] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AB-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1333, avg_group_log_size=121) [2025-01-08 00:50:49.453687] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AB-0-0] [lt=139] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=10, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:49.453701] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AB-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=47522, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:49.453710] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AB-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17770, log_task={header:{begin_lsn:{lsn:2162879}, end_lsn:{lsn:2163001}, log_id:17770, min_scn:{val:1736326249201328001, v:0}, max_scn:{val:1736326249201328001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2162757}, prev_proposal_id:1, committed_end_lsn:{lsn:2162879}, data_checksum:940009244, accum_checksum:2769181528, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326249353154, freeze_ts:1736326249353155, submit_ts:1736326249353260, flushed_ts:1736326249359124, gen_to_freeze cost time:1, gen_to_submit cost time:106, submit_to_flush cost time:5864}, fs_cb_begin_ts=1736326249453679, log_life_time=100525) [2025-01-08 00:50:49.459414] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.459415] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=75] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=656101376, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=417640448, wash_ratio=6, tenant_id=1001) [2025-01-08 00:50:49.459543] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=38] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:50:49.459625] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=78] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2790, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:49.459694] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1110196224, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2111029248, wash_ratio=6, tenant_id=1002) [2025-01-08 00:50:49.459750] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:50:49.459764] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:49.459870] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.468693] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-01-08 00:50:49.469054] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43E9-0-0] [lt=31] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:49.470653] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:49.471890] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58289][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=248] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:50:49.483532] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=81] PNIO [ratelimit] time: 1736326249483529, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007629, add_bytes: 0 [2025-01-08 00:50:49.495694] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.517329] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=64] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1923, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:49.517384] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4076) [2025-01-08 00:50:49.517412] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=78, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:49.517449] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:49.519512] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A7F-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=25, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:49.519549] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A7F-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4208, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:49.519563] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A7F-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=94, avg_log_submit_to_flush_time=2177, avg_log_submit_to_slide_time=4113) [2025-01-08 00:50:49.519793] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4076, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.519831] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=36] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.527624] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.529633] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=21, avg_log_batch_cnt=1, total_group_log_size=10530, avg_group_log_size=501) [2025-01-08 00:50:49.529662] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10530) [2025-01-08 00:50:49.541623] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:49.541671] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:50:49.541692] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1332) [2025-01-08 00:50:49.541706] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:49.542056] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=188] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.542516] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=46] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:50:49.542683] INFO [STORAGE] try_shrink (ob_compaction_memory_pool.cpp:522) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=39] Compaction Mem Pool current stat: (ret=0, max_block_num=64, total_block_num=8, Chunk list size: =1) [2025-01-08 00:50:49.544242] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:50:49.544506] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.545352] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82EB-0-0] [lt=53] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=97, avg_log_submit_to_flush_time=5319, avg_log_submit_to_slide_time=6495) [2025-01-08 00:50:49.545566] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F32-0-0] [lt=117] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:49.545812] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F32-0-0] [lt=244] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4716, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:49.548803] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=132] no need to update min/max version(new_version={val:1736326249147402000, v:0}, min_version={val:1736326249147402000, v:0}, max_version={val:1736326249147402000, v:0}) [2025-01-08 00:50:49.548832] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:49.554663] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ABD5-0-0] [lt=158] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=108) [2025-01-08 00:50:49.555515] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:49.557903] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=17] [APPLY STAT CB APPEND COST TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=152, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:50:49.557928] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=25] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=5939, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:50:49.557937] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:50:49.557945] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=-1045, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:50:49.557953] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=5046, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:50:49.561032] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:50:49.563520] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:49.579634] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:49.595950] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.621491] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=15] [APPLY STAT CB APPEND COST TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=156, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:50:49.621528] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=37] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=3970, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:50:49.621537] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:50:49.621698] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=159] [APPLY STAT CB EXECUTE TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=-779, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:50:49.621707] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB TOTAL TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=3347, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:50:49.643060] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=205] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3198, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:49.643125] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=269, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:49.646622] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=124, this=0x2ad934926280, extra_info="") [2025-01-08 00:50:49.646665] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=43] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=5538, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:50:49.646677] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:50:49.646685] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=-784, this=0x2ad934926910, extra_info="") [2025-01-08 00:50:49.646692] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=4879, this=0x2ad934926b40, extra_info="") [2025-01-08 00:50:49.646861] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:49.651478] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AF-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17772, log_task={header:{begin_lsn:{lsn:2163123}, end_lsn:{lsn:2163245}, log_id:17772, min_scn:{val:1736326249201328003, v:0}, max_scn:{val:1736326249201328003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2163001}, prev_proposal_id:1, committed_end_lsn:{lsn:2163123}, data_checksum:940009244, accum_checksum:1691600049, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326249551452, freeze_ts:1736326249551452, submit_ts:1736326249551461, flushed_ts:1736326249555884, gen_to_freeze cost time:0, gen_to_submit cost time:9, submit_to_flush cost time:4423}, fs_cb_begin_ts=1736326249651468, log_life_time=100016) [2025-01-08 00:50:49.661872] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:50:49.666225] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58301][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=13] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:50:49.668868] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43ED-0-0] [lt=75] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:49.670275] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1558, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:49.670317] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:49.670341] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=128, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:49.670403] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=61, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:49.691140] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:50:49.696103] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.728395] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=113] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.728735] INFO [COMMON] wash (ob_kvcache_store.cpp:356) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=48] Wash time detail, (compute_wash_size_time=147, refresh_score_time=49, wash_time=0) [2025-01-08 00:50:49.756524] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2750, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:49.756563] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=308, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:49.756653] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=45, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:49.761534] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:50:49.766291] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=31] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:49.769418] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43EF-0-0] [lt=103] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:50:49.773388] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=90, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:49.773430] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:49.774933] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:49.790403] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=92] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1001, is_allowed_broadcast=true) [2025-01-08 00:50:49.796225] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.800623] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58299][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B45-0-0] [lt=35] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10537) [2025-01-08 00:50:49.805607] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=38] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:49.809061] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49CFFF-0-0] [lt=42][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.816673] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=77, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:49.834591] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1736326249834587, bytes: 866070813, bw: 0.218721 MB/s, add_ts: 1004177, add_bytes: 230304 [2025-01-08 00:50:49.844161] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1755) [2025-01-08 00:50:49.848565] INFO [STORAGE.COMPACTION] add_event_and_diagnose (ob_medium_loop.cpp:203) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=100] finish schedule all tablet merge(merge_version_=1, schedule_stats_={all_ls_weak_read_ts_ready:true, start_timestamp:0}, loop_cnt=0, tenant_merge_finish=true, is_scan_all_tablet_finish=true, schedule_tablet_cnt={schedule_dag_cnt:0, submit_clog_cnt:0, finish_cnt:1105, loop_tablet_cnt:1105, force_freeze_cnt:0, wait_rs_validate_cnt:0}, time_guard=GET_TABLET=20.27ms|INIT_TABLET_STATUS=89.54ms|SCHEDULE_NEXT_MEDIUM=7.62ms|SCHEDULE_TABLET_EXECUTE=819.37ms|) [2025-01-08 00:50:49.848620] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:95) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=49] MediumLoopTask(cost_ts=937129) [2025-01-08 00:50:49.851827] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5180384760, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:49.851863] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:49.851876] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8bf5f0530) [2025-01-08 00:50:49.851885] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5180384760, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:49.855828] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C81-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.856444] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1}, {id:1002}]) [2025-01-08 00:50:49.857719] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F85790-0-0] [lt=47] MediumCheckTask(cost_ts=8) [2025-01-08 00:50:49.858049] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:49.864518] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C81-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.888760] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=80] ====== checkpoint timer task ====== [2025-01-08 00:50:49.889867] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=74] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:49.890051] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=181] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:49.890301] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=115] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259254996000, latest_log_ts:{val:1736326242681587001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:49.890484] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=176] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259254996000, latest_log_ts:{val:1736326242681587001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:50:49.890637] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=143] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:49.890653] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1736326242681587001, v:0}) [2025-01-08 00:50:49.890662] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:50:49.890701] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:50:49.890771] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=57] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:49.890855] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=64] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:49.890892] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=30] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:49.890930] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:49.890977] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=45] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:49.890992] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:50:49.896592] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.897348] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.897406] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.897476] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=113] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.897509] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.905831] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:49.929393] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A88-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:49.929718] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.930365] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=255] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=93, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:49.942389] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:49.946663] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=151] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12816728064, fast_delete_log_mode=false) [2025-01-08 00:50:49.946747] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=7] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:50:49.947152] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:50:49.947401] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=26, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:49.947577] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F3E-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1877) [2025-01-08 00:50:49.951868] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:49.953286] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:49.954073] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:49.954498] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58207][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=25] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:50:49.955136] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58337][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:49.957437] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=163] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:50:49.961694] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:50:49.973504] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58337][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D600-0-0] [lt=27] init point thread id with(&point=0x5572d50e2180, idx_=993, point=[thread id=58337, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58337) [2025-01-08 00:50:49.973659] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17775, miss_cnt:0, cache_read_size:2174311, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:49.980597] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=68] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8549, offset=2424795}, end_cursor:ObLogCursor{file_id=1, log_id=8550, offset=2425100}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894684050, len:4133, log_data_len:305, seq:8549, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:49.981029] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3B-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:49.981078] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3B-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326249904148000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326249904148000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:49.993015] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1736326249989823000, start_id=1736326249989823000, end_id=1736326249989823001) [2025-01-08 00:50:49.995566] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=37] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8550, offset=2425100}, end_cursor:ObLogCursor{file_id=1, log_id=8551, offset=2425397}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894688050, len:3828, log_data_len:297, seq:8550, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:49.996200] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3B-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:49.996295] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3B-0-0] [lt=73] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326249904148000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326249904148000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:49.996871] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.998389] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:49.998433] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:1047, bw:4287769, [delay/us]:prepare:1, schedule:189, submit:0, rt:350, total:541) [2025-01-08 00:50:49.998457] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, bw:40835, [delay/us]:prepare:11, schedule:422, submit:131, rt:1145, total:1709) [2025-01-08 00:50:49.998524] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=58] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=261, ops=2, iops=263, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:49.998618] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=87] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1047, avg_bw:4287585, [delay/us]: prepare:1, schedule:189, submit:0, device:350, total:541) [2025-01-08 00:50:49.998637] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40834, [delay/us]: prepare:11, schedule:422, submit:131, device:1145, total:1709) [2025-01-08 00:50:49.998649] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:49.998690] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4527, iops:19, bw:85754, [delay/us]:prepare:7, schedule:762, submit:348, rt:1672, total:2791) [2025-01-08 00:50:49.998701] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:49.998739] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4527, avg_iops:19, avg_bw:85765, [delay/us]: prepare:7, schedule:762, submit:348, device:1672, total:2791) [2025-01-08 00:50:49.998750] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:49.998771] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:6144, iops:2, bw:12252, [delay/us]:prepare:7, schedule:131, submit:1, rt:1148, total:1288) [2025-01-08 00:50:49.998793] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4193, iops:42, bw:175622, [delay/us]:prepare:6, schedule:344, submit:184, rt:1466, total:2002) [2025-01-08 00:50:49.998802] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:49.998879] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=71] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:6144, avg_iops:2, avg_bw:12254, [delay/us]: prepare:7, schedule:131, submit:1, device:1148, total:1288) [2025-01-08 00:50:49.998896] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4194, avg_iops:42, avg_bw:175644, [delay/us]: prepare:6, schedule:344, submit:184, device:1466, total:2002) [2025-01-08 00:50:49.998906] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:49.998927] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:2332, total:2341) [2025-01-08 00:50:49.998952] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:49.998980] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:2332, total:2341) [2025-01-08 00:50:49.998997] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:49.999012] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:49.999021] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:49.999029] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:49.999038] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=227kB/s, net_out=213kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:50.009936] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=8] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:50:50.010034] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=88] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:50:50.010052] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=14] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:50:50.010075] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=21] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:50.011739] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28E-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:50.011785] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28E-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326249485990004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326249485990004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:50.014422] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=94] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:50:50.016327] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=47] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:50:50.018679] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=43] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:50:50.018719] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:50:50.020145] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:50:50.020287] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=39] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:50:50.020308] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=18] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:50:50.023285] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B89-0-0] [lt=23] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:50.025316] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=13] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:50:50.026273] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A84-0-0] [lt=37] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3661) [2025-01-08 00:50:50.027506] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=46] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:50:50.029183] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=47] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:50:50.029462] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=276] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:50:50.031522] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:50:50.031611] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=38] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:50:50.031626] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=13] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:50:50.034609] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42999713}], scn_array=[cnt:1, {val:1736326250001028000, v:0}], curr_size=749, accum_size=7118, time_cost=29045) [2025-01-08 00:50:50.035456] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84157402}], scn_array=[cnt:1, {val:1736326249904148000, v:0}], curr_size=122, accum_size=1537, time_cost=11811) [2025-01-08 00:50:50.038418] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=11] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:50:50.040276] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=107] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:50.040978] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=40] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:50:50.042802] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=26] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:50:50.042842] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:50:50.044456] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:50:50.044546] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=39] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:50:50.044563] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:50:50.044585] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=12] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:50:50.045289] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=700] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:50.045344] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=45] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:50:50.045355] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:50.045376] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:50:50.045395] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=19] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:50:50.045406] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=7] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:50:50.045412] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:50.045428] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:50:50.045439] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=10] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:50:50.045447] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=4] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:50:50.045452] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=5] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:50.045466] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=13] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:50:50.045477] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=11] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:50:50.046759] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=12] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:50:50.046799] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=37] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:50:50.046811] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=10] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:50:50.046847] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=8] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:50:50.046860] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:50:50.046869] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:50:50.046881] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:50:50.046896] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=14] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:50:50.047143] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=1877, avg_group_log_size=156) [2025-01-08 00:50:50.047278] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1877) [2025-01-08 00:50:50.048846] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=185] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1984, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:50.048982] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=129] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=611, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:50.049022] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:50.049054] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=41] Slow write(duration=10471, write_len=4096) [2025-01-08 00:50:50.049069] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=14] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8551, offset=2425397}, end_cursor:ObLogCursor{file_id=1, log_id=8552, offset=2425694}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89468c050, len:3531, log_data_len:297, seq:8551, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:50.051906] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=16] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8552, offset=2425694}, end_cursor:ObLogCursor{file_id=1, log_id=8553, offset=2425991}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894690050, len:3234, log_data_len:297, seq:8552, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:50.052298] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36B7-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:50:50.052551] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17687, miss_cnt:0, cache_read_size:2163733, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:50.052913] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1454, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.055819] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:50.063862] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:50:50.080407] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=90] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:50.096965] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.130799] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6752137216, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.147046] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:50.153520] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36B9-0-0] [lt=24] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=17, avg_log_submit_to_flush_time=3512, avg_log_submit_to_slide_time=56682) [2025-01-08 00:50:50.153563] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36B9-0-0] [lt=41][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17777, log_task={header:{begin_lsn:{lsn:2163733}, end_lsn:{lsn:2163855}, log_id:17777, min_scn:{val:1736326249699974003, v:0}, max_scn:{val:1736326249699974003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2163611}, prev_proposal_id:1, committed_end_lsn:{lsn:2163733}, data_checksum:940009244, accum_checksum:2879697556, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326250052281, freeze_ts:1736326250052281, submit_ts:1736326250052288, flushed_ts:1736326250056122, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:3834}, fs_cb_begin_ts=1736326250153510, log_life_time=101229) [2025-01-08 00:50:50.153684] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=152] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.157699] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ABDE-0-0] [lt=78] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.766940498352050781e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=388, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.714020632270641045e+02) [2025-01-08 00:50:50.158254] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:50.161839] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:3652K/907:229920 disk:3652K/907:229920 t1m2:3368K/841:205468 t1m2g0:3368K/841:205468 t1002m2:196K/47:439630 t1002m2g0:196K/47:439630 t1001m2:88K/19:768426 t1001m2g0:88K/19:768426 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:50.162662] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58339][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.162712] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:50:50.164475] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get_max_applied_scn(scn={val:1736326249904148001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2141388}, palf_committed_end_scn_:{val:1736326249904148002, v:0}, last_check_scn_:{val:1736326249904148002, v:0}, max_applied_cb_scn_:{val:1736326249904148001, v:0}}) [2025-01-08 00:50:50.164514] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2141388}, readable_log_tail:{lsn:2141388}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64963380, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:50.164671] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:50:50.170194] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC368-0-0] [lt=50][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.172385] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43F7-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=36, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:50.172425] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43F7-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12954, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:50.172438] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43F7-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=36, avg_log_submit_to_flush_time=2147, avg_log_submit_to_slide_time=12918) [2025-01-08 00:50:50.172550] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.175943] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A46-0-0] [lt=102] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.187324] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB023-0-0] [lt=86] get number(ret=0, service_type_=0, range=1, base_id=1736326250187003000, start_id=1736326250187003000, end_id=1736326250187003001) [2025-01-08 00:50:50.188853] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=28] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:50.190346] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=99] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:50.190462] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=41] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326244258401000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:50.190518] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=54] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326244258401000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326244258401000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:50.196573] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1609BCBB-0-0] [lt=146] get number(ret=0, service_type_=0, range=1, base_id=1736326250196395000, start_id=1736326250196395000, end_id=1736326250196395001) [2025-01-08 00:50:50.197245] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.198405] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58340][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.199463] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:50:50.203530] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE7C9-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1736326250202295000, start_id=1736326250202295000, end_id=1736326250202295001) [2025-01-08 00:50:50.211619] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A48-0-0] [lt=23] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.214782] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=80] gc stale ls task succ [2025-01-08 00:50:50.221130] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:50.221182] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:50.221798] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB44-0-0] [lt=24][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193755135630, cache_obj->added_lc()=false, cache_obj->get_object_id()=13551, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:50.224889] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A49-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.227145] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4A-0-0] [lt=82] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.228684] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=3661, avg_group_log_size=332) [2025-01-08 00:50:50.228770] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=81] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3661) [2025-01-08 00:50:50.228790] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:50.230511] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8700D-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.230575] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:50.230589] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:50.236542] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4B-0-0] [lt=16] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.242468] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=29] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:50:50.242511] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=38] start lib cache evict(tenant_id=1, mem_hold=93765632, mem_limit=107374180, cache_obj_num=300, cache_node_num=300) [2025-01-08 00:50:50.242525] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=93765632, mem_limit=107374180, cache_obj_num=300, cache_node_num=300) [2025-01-08 00:50:50.242537] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:50.245449] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=45] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:50.245453] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=300, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}]) [2025-01-08 00:50:50.246297] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=797] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:50.246636] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=29] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:50.246731] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245675194000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:50.247161] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1609BCBB-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1736326250241481000, start_id=1736326250241481000, end_id=1736326250241481001) [2025-01-08 00:50:50.262385] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:50.279631] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C98-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:50.279670] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C98-0-0] [lt=36] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:50.279707] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=13] gc stale ls task succ [2025-01-08 00:50:50.291974] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540D-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.298382] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.300066] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4C-0-0] [lt=102] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.312555] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4D-0-0] [lt=46] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.319420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.319463] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.329973] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=36] current monitor number(seq_=-1) [2025-01-08 00:50:50.331314] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.331716] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4E-0-0] [lt=95] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.333115] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=24] failed to get result(ret=-4008) [2025-01-08 00:50:50.333408] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=16] check new jobs(ret=0, tenant_id=1001, is_oracle_tenant=false, job_infos=[{tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326185605925, start_date_:1736325283345722, last_date_:1736326183353587, this_date_:0, next_date_:1736327083345722, end_date_:64060560000000000, total_:6596354, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736410783345628, last_date_:0, this_date_:0, next_date_:1736410783345628, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:50:50.333690] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=278] check all tenants(ret=0) [2025-01-08 00:50:50.339539] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A50-0-0] [lt=9] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.341507] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=66] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:50.342681] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=49] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:50.342793] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=39] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245564663000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:50.342821] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326245564663000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326245564663000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:50.343136] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD84-0-0] [lt=28] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326250435371, keepalive_ts:-1}) [2025-01-08 00:50:50.343162] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD84-0-0] [lt=24] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326250435371, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:50.347682] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=77, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:50.353782] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:50:50.365662] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18639483}], scn_array=[cnt:1, {val:1736326249904148004, v:0}], curr_size=122, accum_size=610, time_cost=11047) [2025-01-08 00:50:50.368137] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2164099}], scn_array=[cnt:1, {val:1736326250308616000, v:0}], curr_size=122, accum_size=610, time_cost=13288) [2025-01-08 00:50:50.368275] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:50:50.369226] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4120, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:50.369255] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9757) [2025-01-08 00:50:50.369288] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=542, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:50.369368] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=24, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:50.371831] INFO [LIB] stat (utility.h:1210) [58241][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37E7-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=11, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:50.371855] INFO [LIB] stat (utility.h:1210) [58241][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37E7-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=6078, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:50.371865] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58241][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37E7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=118, avg_log_submit_to_flush_time=4864, avg_log_submit_to_slide_time=5832) [2025-01-08 00:50:50.373008] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1B5BBA44-0-0] [lt=30] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:50.373280] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E4-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:50.373321] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E4-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326250308615999, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326250308615999, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:50.376522] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9757, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.376891] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=76] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326249699974003, v:0}, new_max_version={val:1736326249699974003, v:0}, last_min_version={val:1736326249404190004, v:0}, last_max_version={val:1736326249404190004, v:0}, total_time_us=11072, delta_us=676914) [2025-01-08 00:50:50.376921] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326249699974003, v:0}, min_version={val:1736326249699974003, v:0}, max_version={val:1736326249699974003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:50.381372] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:50:50.381375] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58341][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.386163] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=49] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:50.389948] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:50.389988] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-01-08 00:50:50.390000] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:50.390015] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:50.390030] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193754303479, flush_io_finished_ret_:0, flush_io_finished_round_:1909, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:50.391090] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=9] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:50.391140] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=37] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:50.391182] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=16] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=109) [2025-01-08 00:50:50.393024] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37E8-0-0] [lt=23] init point thread id with(&point=0x5572d50e2280, idx_=997, point=[thread id=58341, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58341) [2025-01-08 00:50:50.394925] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:50.394958] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-01-08 00:50:50.394968] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:50.394982] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:50.394997] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193754307012, flush_io_finished_ret_:0, flush_io_finished_round_:3737, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:50.395108] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:50.398589] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.403925] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=176] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:50.404027] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=88] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:50.404043] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:50:50.404050] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:50:50.404111] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=5] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:50:50.404130] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=17] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=94) [2025-01-08 00:50:50.407162] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=132] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=503, get_gts_with_stc_cnt=32, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=30, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:50.407366] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=133] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=532, get_gts_with_stc_cnt=73, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=73, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:50.433458] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58342][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.433611] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:50:50.452257] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:50.452783] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:50.452841] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2164277}, scn={val:1736326250308616001, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:50:50.452915] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=51] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:16, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326250452692, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2164333}, next_to_submit_scn_:{val:1736326250308616002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2164221}, end_lsn:{lsn:2164333}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2164333}, readable_log_tail:{lsn:2164333}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64940435, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326250308616001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326250308616001, v:0}, data_checksum:2012734446, flag:54327}}, init_mode_version:0, accumulate_checksum:2912715304, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326250308616001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17692, miss_cnt:0, cache_read_size:2164333, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2164277}, scn_:{val:1736326250308616001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326250364376, first_handle_ts_:1736326250452913, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad94da04150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:50:50.453223] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324445564663000) [2025-01-08 00:50:50.453299] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=73] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2164277}, scn_:{val:1736326250308616001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326250364376, first_handle_ts_:1736326250453206, replay_cost_:90, retry_cost_:-1, read_log_buf_:0x2ad94da04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326250452692, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2164333}, next_to_submit_scn_:{val:1736326250308616002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2164221}, end_lsn:{lsn:2164333}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2164333}, readable_log_tail:{lsn:2164333}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64940435, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326250308616001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326250308616001, v:0}, data_checksum:2012734446, flag:54327}}, init_mode_version:0, accumulate_checksum:2912715304, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326250308616001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17692, miss_cnt:0, cache_read_size:2164333, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=24, ret=0) [2025-01-08 00:50:50.454684] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=28] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:50.455444] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:50.458352] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BEC1-0-0] [lt=28] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:50:50.463176] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.463263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.464793] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8305-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:50.465323] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8305-0-0] [lt=527] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7592, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:50.465688] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.467454] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58342][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A89-0-0] [lt=128] init point thread id with(&point=0x5572d50e22c0, idx_=998, point=[thread id=58342, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58342) [2025-01-08 00:50:50.468596] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F4F-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:50.468620] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F4F-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=46249, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:50.474856] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43FB-0-0] [lt=41] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:50.491181] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1736326250491177, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007648, add_bytes: 0 [2025-01-08 00:50:50.491668] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D22484647-0-0] [lt=36] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:50:50.498724] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.520007] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.531708] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.533406] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=9757, avg_group_log_size=542) [2025-01-08 00:50:50.533438] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9757) [2025-01-08 00:50:50.536313] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58344][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.536379] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] worker thread created(tenant_->id()=1001, group_id_=1, token=8) [2025-01-08 00:50:50.542356] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=253] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.544646] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.548283] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:50.548315] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:50.548518] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=198] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:50.548559] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:50.550622] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:50.551424] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F51-0-0] [lt=46] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=112, avg_log_submit_to_flush_time=5820, avg_log_submit_to_slide_time=7442) [2025-01-08 00:50:50.552142] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F52-0-0] [lt=48] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:50.552241] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F52-0-0] [lt=97] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6361, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:50.553306] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36C1-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=12, avg_log_batch_cnt=0, total_group_log_size=1454, avg_group_log_size=121) [2025-01-08 00:50:50.556058] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:50.562004] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCD-0-0] [lt=25] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:50.562052] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCD-0-0] [lt=40] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=52) [2025-01-08 00:50:50.569140] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:50:50.569346] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:50.571453] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58344][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B53-0-0] [lt=38] init point thread id with(&point=0x5572d50e2340, idx_=1000, point=[thread id=58344, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58344) [2025-01-08 00:50:50.575795] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=91] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:50:50.575823] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=1) [2025-01-08 00:50:50.575834] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:50:50.575844] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:50:50.580607] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=50] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:50.598880] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.616404] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B8B-0-0] [lt=34] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:50.616566] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B8D-0-0] [lt=37] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:50.631534] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3242, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:50.631593] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=56] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3661) [2025-01-08 00:50:50.631620] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=250, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:50.631654] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=30, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:50.632415] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A8A-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:50.632442] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A8A-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4978, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:50.632453] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A8A-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=150, avg_log_submit_to_flush_time=3581, avg_log_submit_to_slide_time=4827) [2025-01-08 00:50:50.632558] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3661, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.647289] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:50.649841] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=139] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2985, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:50.649940] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=346, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:50.665950] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:50.673924] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43FF-0-0] [lt=52] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1089) [2025-01-08 00:50:50.674936] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=1509, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:50.674966] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1089) [2025-01-08 00:50:50.675013] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=254, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:50.675247] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=67, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:50.685789] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [46340][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=19] XA scheduler heartbeat task statistics(avg_time=0) [2025-01-08 00:50:50.696506] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A2-0-0] [lt=14] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250687847, keepalive_ts:-1}) [2025-01-08 00:50:50.696621] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A2-0-0] [lt=112] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250687847, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:50.696766] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A3-0-0] [lt=21] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250688003, keepalive_ts:-1}) [2025-01-08 00:50:50.696803] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A3-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250688003, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:50.699082] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.732224] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=16] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.734569] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=45] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:50:50.760768] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=34] ====== tenant freeze timer task ====== [2025-01-08 00:50:50.766376] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:50.769453] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=155] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:50.774357] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4401-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1089, avg_group_log_size=121) [2025-01-08 00:50:50.775584] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=39] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:50.792901] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D002-0-0] [lt=120][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.799227] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.800444] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37EF-0-0] [lt=150] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10347) [2025-01-08 00:50:50.802148] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58342][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0EE-0-0] [lt=34] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250790230, keepalive_ts:-1}) [2025-01-08 00:50:50.802182] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58342][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0EE-0-0] [lt=33] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250790230, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:50.805830] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=98] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:50.809100] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=57] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:50.829587] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=99, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:50.835423] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A6E-0-0] [lt=128] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326250835333}) [2025-01-08 00:50:50.835466] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594E-0-0] [lt=27] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326250835444, sql_renew_time:1736326250835444, is_mark_fail:false}]) [2025-01-08 00:50:50.835506] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594E-0-0] [lt=39] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326250835333}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326250835444, sql_renew_time:1736326250835444, is_mark_fail:false}]) [2025-01-08 00:50:50.839867] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1736326250839864, bytes: 866403270, bw: 0.315391 MB/s, add_ts: 1005277, add_bytes: 332457 [2025-01-08 00:50:50.850090] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:50.852656] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30] start to print blacklist info [2025-01-08 00:50:50.853019] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=24] ls blacklist refresh finish(cost_time=19770) [2025-01-08 00:50:50.853084] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5181673537, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:50.853114] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:50.853127] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a0110) [2025-01-08 00:50:50.853136] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5181673537, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:50.855550] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=108] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3110, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:50.855601] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1048, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:50.855689] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=76, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:50.855927] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=111] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=210, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:50.855964] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:50.856618] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}, {id:1003}, {id:1001}, {id:1002}]) [2025-01-08 00:50:50.861209] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326250662489000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=198638, in_cluster_service=true, cluster_version={val:1736326250456027000, v:0}, min_cluster_version={val:1736326250456027000, v:0}, max_cluster_version={val:1736326250456027000, v:0}, get_cluster_version_err=0, cluster_version_delta=405100, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36255, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=505358114, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:50.874107] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C82-0-0] [lt=103][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.879154] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4403-0-0] [lt=40][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17870, log_task={header:{begin_lsn:{lsn:2175034}, end_lsn:{lsn:2175156}, log_id:17870, min_scn:{val:1736326250487243002, v:0}, max_scn:{val:1736326250487243002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2174912}, prev_proposal_id:1, committed_end_lsn:{lsn:2175034}, data_checksum:1715074492, accum_checksum:3397965136, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326250774283, freeze_ts:1736326250774283, submit_ts:1736326250774291, flushed_ts:1736326250777045, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:2754}, fs_cb_begin_ts=1736326250879143, log_life_time=104860) [2025-01-08 00:50:50.881105] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C82-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.885034] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=93] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:50:50.893450] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=34] ====== check clog disk timer task ====== [2025-01-08 00:50:50.893491] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=34] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:50:50.897215] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.897260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.897596] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.897750] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.899414] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.908112] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326250908032], gts:1736326250907258000, latest_srr:[mts=1736326250908032]}) [2025-01-08 00:50:50.908159] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=45] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:50.908236] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=57] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326250908218], gts:1736326250907258000, latest_srr:[mts=1736326250908218]}) [2025-01-08 00:50:50.908568] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326250908553], gts:1736326250908389000, latest_srr:[mts=1736326250908553]}) [2025-01-08 00:50:50.908991] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=17] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18640215}, readable_log_tail:{lsn:18640215}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48464553, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:50.909097] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=105] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:50.909112] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=13] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2142242}, readable_log_tail:{lsn:2142242}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64962526, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:50.909292] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=178] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:50.909309] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12] get_replay_process success(id={id:1002}, submitted_log_size=2175156, unsubmitted_log_size=2175156, replayed_log_size=2175156, unreplayed_log_size=0) [2025-01-08 00:50:50.909321] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=8] get_replay_process success(id={id:1003}, submitted_log_size=2164943, unsubmitted_log_size=2164943, replayed_log_size=2164943, unreplayed_log_size=0) [2025-01-08 00:50:50.909350] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=7] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:50:50.926282] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] ====== check clog disk timer task ====== [2025-01-08 00:50:50.926316] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=28] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:50.932685] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.948751] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:50.951527] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:50:50.952132] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:50.952259] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:50.952944] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:50.954140] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1454) [2025-01-08 00:50:50.954535] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:50.955723] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:50.955830] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58298][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=136] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:50:50.970664] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1069504, replace_num=15728) [2025-01-08 00:50:50.970754] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=88] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:50.977651] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17784, miss_cnt:0, cache_read_size:2175400, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:50.994161] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D2E-0-0] [lt=18] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:50.996991] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=86, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:50.999548] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.001453] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:51.001520] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44917, [delay/us]:prepare:8, schedule:109, submit:1612, rt:1304, total:3035) [2025-01-08 00:50:51.001537] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:51.001592] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44923, [delay/us]: prepare:8, schedule:109, submit:1612, device:1304, total:3035) [2025-01-08 00:50:51.001605] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:51.001644] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4864, iops:16, bw:77594, [delay/us]:prepare:26, schedule:135, submit:589, rt:3187, total:3938) [2025-01-08 00:50:51.001654] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:51.001689] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4864, avg_iops:16, avg_bw:77595, [delay/us]: prepare:26, schedule:135, submit:589, device:3187, total:3938) [2025-01-08 00:50:51.001700] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:51.001722] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:2, bw:8167, [delay/us]:prepare:4, schedule:4196, submit:1, rt:1962, total:6164) [2025-01-08 00:50:51.001744] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:171525, [delay/us]:prepare:7, schedule:149, submit:595, rt:1493, total:2245) [2025-01-08 00:50:51.001753] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:51.001780] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:2, avg_bw:8168, [delay/us]: prepare:4, schedule:4196, submit:1, device:1962, total:6164) [2025-01-08 00:50:51.001796] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:171534, [delay/us]: prepare:7, schedule:149, submit:595, device:1493, total:2245) [2025-01-08 00:50:51.001806] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:51.001824] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1006, total:1017) [2025-01-08 00:50:51.001951] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=105] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:51.001991] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4083, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1006, total:1017) [2025-01-08 00:50:51.002011] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:51.002029] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:51.002038] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:51.002045] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:51.002053] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=316kB/s, net_out=255kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:51.031130] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3540) [2025-01-08 00:50:51.048653] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28F-0-0] [lt=117] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:51.048723] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28F-0-0] [lt=50] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326250487243004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326250487243004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:51.048823] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:51.048850] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:51.051123] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=91] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3897, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:51.051178] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=995, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:51.051341] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=228] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:51.054622] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.054644] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36CB-0-0] [lt=51] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:50:51.058311] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:51.058949] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8553, offset=2425991}, end_cursor:ObLogCursor{file_id=1, log_id=8554, offset=2426296}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894694050, len:2937, log_data_len:305, seq:8553, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:51.059590] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3C-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:51.059637] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3C-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326250908389000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326250908389000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:51.066394] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C28F-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1736326251060618000, start_id=1736326251060618001, end_id=1736326251060618002) [2025-01-08 00:50:51.070146] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:51.072241] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8554, offset=2426296}, end_cursor:ObLogCursor{file_id=1, log_id=8555, offset=2426593}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894698050, len:2632, log_data_len:297, seq:8554, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:51.072673] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3C-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:51.072793] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3C-0-0] [lt=104] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326250908389000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326250908389000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:51.080802] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=41] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:51.086037] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8555, offset=2426593}, end_cursor:ObLogCursor{file_id=1, log_id=8556, offset=2426890}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89469c050, len:2335, log_data_len:297, seq:8555, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:51.087892] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8556, offset=2426890}, end_cursor:ObLogCursor{file_id=1, log_id=8557, offset=2427187}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946a0050, len:2038, log_data_len:297, seq:8556, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:51.088069] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=123] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17698, miss_cnt:0, cache_read_size:2165065, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:51.099670] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.100551] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B5BBA47-0-0] [lt=80] get number(ret=0, service_type_=0, range=1, base_id=1736326251099248000, start_id=1736326251099248000, end_id=1736326251099248001) [2025-01-08 00:50:51.133265] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=65] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.134874] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=79] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=377484, clean_num=31457) [2025-01-08 00:50:51.147695] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:51.154037] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36CD-0-0] [lt=31] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=17, avg_log_submit_to_flush_time=4570, avg_log_submit_to_slide_time=31697) [2025-01-08 00:50:51.154152] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.155082] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=96] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2169283, this={dio_aligned_buf:{buf_write_offset:2621, buf_padding_size:1475, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:23779, truncate_used_ts:5345}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2166784, buf_len=122, total_write_size=12172, total_write_size_after_dio=421888, ob_pwrite_used_ts=253471, count=100, each_pwrite_cost=2534) [2025-01-08 00:50:51.155708] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=76] ====== check clog disk timer task ====== [2025-01-08 00:50:51.155738] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:51.169009] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC36B-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.170824] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:50:51.174305] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4409-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=15, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:51.174396] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4409-0-0] [lt=90] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=34884, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:51.174384] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.174411] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4409-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=1978, avg_log_submit_to_slide_time=34869) [2025-01-08 00:50:51.174600] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:312K/72:110736 disk:312K/72:110736 t1m2:44K/10:91587 t1m2g0:44K/10:91587 t1002m2:192K/45:124091 t1002m2g0:192K/45:124091 t1001m2:76K/17:86648 t1001m2g0:76K/17:86648 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:51.197270] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AB8F-0-0] [lt=18] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.742941093444824219e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=387, Throttle Trigger(MB)=232, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.809489877078427753e+02) [2025-01-08 00:50:51.197735] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=48] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:51.199724] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.202506] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE7CF-0-0] [lt=57] get number(ret=0, service_type_=0, range=1, base_id=1736326251201587000, start_id=1736326251201587000, end_id=1736326251201587001) [2025-01-08 00:50:51.204103] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE7CD-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326251203917000, start_id=1736326251203917000, end_id=1736326251203917001) [2025-01-08 00:50:51.212746] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D171BBD5D-0-0] [lt=60] get number(ret=0, service_type_=0, range=1, base_id=1736326251211763000, start_id=1736326251211763000, end_id=1736326251211763001) [2025-01-08 00:50:51.214891] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:50:51.222157] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:51.222300] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=140][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:51.227012] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87013-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.227073] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:51.227085] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:51.246186] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB46-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193756160017, cache_obj->added_lc()=false, cache_obj->get_object_id()=13552, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:51.247706] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:51.247731] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=24] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:51.247781] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E1-0-0] [lt=14] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:51.247800] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E1-0-0] [lt=16] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:51.263993] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:50:51.264278] WDIAG read_multi_row (ob_table_access_helper.h:238) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:50:51.265452] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4549, avg_group_log_size=413) [2025-01-08 00:50:51.265485] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4085) [2025-01-08 00:50:51.265503] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:51.266004] INFO delete_row (ob_table_access_helper.h:339) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326251264298) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326251264298) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:50:51.266079] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:50:51.266135] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:50:51.266186] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:50:51.266555] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=17] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:51.278054] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:50:51.278236] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%", tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:50:51.279325] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] no need to update min/max version(new_version={val:1736326250406804004, v:0}, min_version={val:1736326250406804004, v:0}, max_version={val:1736326250406804004, v:0}) [2025-01-08 00:50:51.279959] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:50:51.280104] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%", tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-01-08 00:50:51.281317] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:50:51.281373] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:51.281396] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:50:51.281405] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:51.281418] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:51.281433] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193755193600, flush_io_finished_ret_:0, flush_io_finished_round_:1914, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:51.281519] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C99-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:51.281543] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C99-0-0] [lt=22] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:51.293675] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540E-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.299758] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:51.299795] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.308830] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF25-0-0] [lt=42] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326251301281, keepalive_ts:-1}) [2025-01-08 00:50:51.308975] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF25-0-0] [lt=142] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326251301281, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:51.319534] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.319573] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.330382] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=45] current monitor number(seq_=-1) [2025-01-08 00:50:51.335687] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=94] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.350335] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=84, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:51.353976] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:50:51.370963] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:50:51.374353] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2175822}, scn={val:1736326250987796003, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:50:51.374555] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=46] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324445564663000) [2025-01-08 00:50:51.374612] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=53] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2175822}, scn_:{val:1736326250987796003, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326251423117, first_handle_ts_:1736326251374538, replay_cost_:72, retry_cost_:-1, read_log_buf_:0x2ad90fe04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326251374188, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2175878}, next_to_submit_scn_:{val:1736326250987796004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2175766}, end_lsn:{lsn:2175878}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2175878}, readable_log_tail:{lsn:2175878}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64928890, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326250987796003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326250987796003, v:0}, data_checksum:796858218, flag:63504}}, init_mode_version:0, accumulate_checksum:2096116064, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326250987796003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17788, miss_cnt:0, cache_read_size:2175878, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=13, ret=0) [2025-01-08 00:50:51.383559] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=2168, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:51.383591] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10185) [2025-01-08 00:50:51.383624] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=266, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:51.383787] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=31, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:51.385406] INFO [LIB] stat (utility.h:1210) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37F9-0-0] [lt=68] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=22, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:51.385430] INFO [LIB] stat (utility.h:1210) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37F9-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4323, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:51.385440] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37F9-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=130, avg_log_gen_to_submit_time=211, avg_log_submit_to_flush_time=2773, avg_log_submit_to_slide_time=4112) [2025-01-08 00:50:51.386044] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=47] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10185, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.386171] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326250701549003, v:0}, new_max_version={val:1736326250701549003, v:0}, last_min_version={val:1736326250406804004, v:0}, last_max_version={val:1736326250406804004, v:0}, total_time_us=6706, delta_us=684620) [2025-01-08 00:50:51.386403] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=227] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326250701549003, v:0}, min_version={val:1736326250701549003, v:0}, max_version={val:1736326250701549003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:51.390223] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:51.390275] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=44] temporary file module use no blocks [2025-01-08 00:50:51.390355] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=78] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:51.390428] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=69] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:51.390469] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=40] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193755303903, flush_io_finished_ret_:0, flush_io_finished_round_:1910, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:51.395170] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:51.395252] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=78] temporary file module use no blocks [2025-01-08 00:50:51.395261] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:51.395299] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=36] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:51.395313] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193755308868, flush_io_finished_ret_:0, flush_io_finished_round_:3738, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:51.399901] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.406217] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E5-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:51.406272] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E5-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326251201587001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326251201587001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:51.408926] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326251408655000, start_id=1736326251408655000, end_id=1736326251408655001) [2025-01-08 00:50:51.418229] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5E5-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326251417174000, start_id=1736326251417174000, end_id=1736326251417174001) [2025-01-08 00:50:51.422006] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=13] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:51.430229] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=47] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:50:51.434906] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:50:51.435074] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58342][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=77] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:50:51.453497] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:51.456035] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:51.458637] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:50:51.458743] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=97] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70285300, tenant_id=1001) [2025-01-08 00:50:51.458794] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=41] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:51.459239] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.459278] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.459601] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36D3-0-0] [lt=96] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:51.471923] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=32] ====== tenant freeze timer task ====== [2025-01-08 00:50:51.498928] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=40] PNIO [ratelimit] time: 1736326251498925, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007748, add_bytes: 0 [2025-01-08 00:50:51.499966] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.520132] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.536291] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.539821] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=10186, avg_group_log_size=565) [2025-01-08 00:50:51.539917] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=88] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10186) [2025-01-08 00:50:51.540528] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:51.541494] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58340][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=42] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:50:51.542353] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.544546] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=82] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.550782] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=61, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:51.550858] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:51.550879] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:51.550894] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:51.553471] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:51.553667] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F6F-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:51.553688] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F6F-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4184, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:51.555078] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F70-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:51.555103] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F70-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3582, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:51.555113] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F70-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=19, avg_log_gen_to_submit_time=116, avg_log_submit_to_flush_time=2278, avg_log_submit_to_slide_time=3482) [2025-01-08 00:50:51.555616] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1765, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.555843] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36D5-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:51.555905] INFO [LIB] stat (utility.h:1210) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36D5-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:51.555978] INFO [LIB] stat (utility.h:1210) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36D5-0-0] [lt=72] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31857, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:51.558750] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=39] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:51.570328] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:51.571173] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:50:51.573851] INFO [LIB] stat (utility.h:1210) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4411-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:51.580994] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=122] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:51.584647] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:2176244}, scn={val:1736326251487377001, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326251576313, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2176244}, next_to_submit_scn_:{val:1736326251487377001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2176122}, end_lsn:{lsn:2176244}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2176244}, readable_log_tail:{lsn:2176244}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64928524, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326251487377000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326251487377000, v:0}, data_checksum:3348837161, flag:7917}}, init_mode_version:0, accumulate_checksum:2050177141, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326251487377000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17791, miss_cnt:0, cache_read_size:2176244, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:51.600253] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.632374] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2124, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:51.632433] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=57] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3962) [2025-01-08 00:50:51.632457] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=354, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:51.632544] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=35, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:51.633715] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA672C-0-0] [lt=47] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:51.633736] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA672C-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4963, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:51.633746] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA672C-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=111, avg_log_submit_to_flush_time=3554, avg_log_submit_to_slide_time=4852) [2025-01-08 00:50:51.633826] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=89] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3962, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.647927] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:51.653826] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=82] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2329, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:51.653861] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=246, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:51.654852] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58346][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:51.655366] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:50:51.675143] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1729, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:51.675234] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:50:51.675258] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=134, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:51.675372] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=76, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:51.677234] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58329][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D623-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1454) [2025-01-08 00:50:51.679944] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=175] get_max_applied_scn(scn={val:1736326251408655001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18641614}, palf_committed_end_scn_:{val:1736326251408655002, v:0}, last_check_scn_:{val:1736326251408655002, v:0}, max_applied_cb_scn_:{val:1736326251408655001, v:0}}) [2025-01-08 00:50:51.680003] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=55] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18641614}, readable_log_tail:{lsn:18641614}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48463154, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:51.680112] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=87] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:51.695457] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=26] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:50:51.700376] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.705832] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58346][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F75-0-0] [lt=39] init point thread id with(&point=0x5572d50e23c0, idx_=1002, point=[thread id=58346, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58346) [2025-01-08 00:50:51.736759] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=61] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.766470] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=45] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:51.767294] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:801) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D1718510F-0-0] [lt=86] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, location:{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, location:{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}]) [2025-01-08 00:50:51.771444] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:50:51.772769] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085141-0-0] [lt=26] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:51.774267] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4415-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:50:51.775958] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:51.789612] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D005-0-0] [lt=50][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.800476] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.803139] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3801-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10186) [2025-01-08 00:50:51.806119] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:51.827892] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=100] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:51.827940] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=38] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:51.827916] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=46] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:51.827958] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=190) [2025-01-08 00:50:51.827969] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=46] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=175) [2025-01-08 00:50:51.830342] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=79] LogLoopThread round_cost_time(us)(round_cost_time=14) [2025-01-08 00:50:51.841423] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1736326251841420, bytes: 866618306, bw: 0.204756 MB/s, add_ts: 1001556, add_bytes: 215036 [2025-01-08 00:50:51.853289] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2066) [2025-01-08 00:50:51.856852] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1}, {id:1003}]) [2025-01-08 00:50:51.859409] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1536, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:51.859441] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=118, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:51.859516] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=67, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:51.859907] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326241846965, end_time=1736326251859905, dag_status={}) [2025-01-08 00:50:51.859958] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=46] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.859975] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.859983] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.859991] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.859998] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860005] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860011] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860017] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860121] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=102] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860135] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860148] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860156] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860171] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860180] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860188] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860197] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860446] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=248] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860462] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860471] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860479] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860488] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860496] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860590] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=93] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860603] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860611] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860619] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860627] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860635] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860642] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860649] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860755] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=104] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860778] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860785] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860793] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860800] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860829] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860841] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860849] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860857] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860864] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860872] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860879] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860886] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860893] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860906] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860915] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860923] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860931] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860938] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860946] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860958] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860967] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860975] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860983] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860991] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860998] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861005] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861013] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861025] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861034] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861042] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861049] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861057] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861064] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861076] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861084] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861092] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861100] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861107] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861114] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861124] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861135] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861143] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861151] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861158] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861167] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861174] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861181] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861240] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861253] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861282] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861295] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:50:51.861303] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:50:51.861310] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:50:51.861315] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:50:51.861321] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:50:51.861326] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:50:51.861330] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:50:51.861334] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:50:51.861339] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:50:51.861343] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:50:51.861348] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:50:51.868423] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=79, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:51.871328] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:51.876667] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=128, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:51.877265] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=596] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:51.890990] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C83-0-0] [lt=47][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.892957] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C83-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.896538] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=84] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.896576] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.897368] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.897408] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.900601] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.909678] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=47] handle gts request statistics(total_rt=2941, total_cnt=284, avg_rt=1.031929824561403564e+01, avg_cnt=2.839999999999999858e+01) [2025-01-08 00:50:51.909744] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=51] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:51.913797] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541C-0-0] [lt=50] InfoPoolResizeTask(cost_ts=10) [2025-01-08 00:50:51.924296] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C79C275-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326251923892000, start_id=1736326251923892000, end_id=1736326251923892001) [2025-01-08 00:50:51.937218] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.951473] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:51.953935] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:50:51.954061] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:51.954566] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:51.954743] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:51.956450] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:51.957110] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:51.971609] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:50:51.977735] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17795, miss_cnt:0, cache_read_size:2176732, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:51.999118] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=60, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:52.000855] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:52.000890] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.003994] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:52.004046] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44942, [delay/us]:prepare:7, schedule:61, submit:53, rt:1217, total:1340) [2025-01-08 00:50:52.004062] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:52.004118] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44942, [delay/us]: prepare:7, schedule:61, submit:53, device:1217, total:1340) [2025-01-08 00:50:52.004132] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:52.004235] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4527, iops:19, bw:85799, [delay/us]:prepare:10, schedule:67, submit:124, rt:1058, total:1261) [2025-01-08 00:50:52.004252] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:52.004292] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4527, avg_iops:19, avg_bw:85792, [delay/us]: prepare:10, schedule:67, submit:124, device:1058, total:1261) [2025-01-08 00:50:52.004303] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:52.004328] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16341, [delay/us]:prepare:4, schedule:17, submit:0, rt:1471, total:1494) [2025-01-08 00:50:52.004351] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4195, iops:41, bw:171584, [delay/us]:prepare:7, schedule:85, submit:346, rt:1136, total:1575) [2025-01-08 00:50:52.004359] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:52.004387] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16341, [delay/us]: prepare:4, schedule:17, submit:0, device:1471, total:1494) [2025-01-08 00:50:52.004402] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4196, avg_iops:41, avg_bw:171584, [delay/us]: prepare:7, schedule:85, submit:346, device:1136, total:1575) [2025-01-08 00:50:52.004412] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:52.004432] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4085, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:217, total:226) [2025-01-08 00:50:52.004455] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:52.004481] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:217, total:226) [2025-01-08 00:50:52.004499] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:52.004514] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:52.004522] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:52.004530] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:52.004538] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=206kB/s, net_out=219kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:52.009906] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46506][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB48-0-0] [lt=81][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193756923739, cache_obj->added_lc()=false, cache_obj->get_object_id()=13553, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:52.017600] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB4D-0-0] [lt=193] get number(ret=0, service_type_=0, range=1, base_id=1736326252017250000, start_id=1736326252017250000, end_id=1736326252017250001) [2025-01-08 00:50:52.022009] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:50:52.036969] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3E-0-0] [lt=44] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326252138826, keepalive_ts:-1}) [2025-01-08 00:50:52.037070] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3E-0-0] [lt=99] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326252138826, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:52.051541] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=2066, avg_group_log_size=206) [2025-01-08 00:50:52.051587] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2066) [2025-01-08 00:50:52.053837] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2073, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:52.053874] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=118, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:52.054661] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:52.055880] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36DF-0-0] [lt=121] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:52.056359] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:52.059001] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:52.067431] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=36] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:52.067468] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=103) [2025-01-08 00:50:52.070638] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4949) [2025-01-08 00:50:52.070719] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=110] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:52.076615] INFO [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1165) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] refresh queuing mode(ret=0, tenant_id=1002, stream_cnt=0, update_schema_cnt=0, cost_time=11) [2025-01-08 00:50:52.076642] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1002, interval_step=1, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1002, bucket_cnt:0, tenant_mem_user_limit:3221225472, tenant_mem_user_hold:243437568, cached_node_array:0x2ad8ba632ee0}) [2025-01-08 00:50:52.079679] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C290-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:52.079781] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C290-0-0] [lt=45] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326251986852000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326251986852000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:52.081232] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:52.088012] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185781-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193757001844, cache_obj->added_lc()=false, cache_obj->get_object_id()=13554, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:52.088518] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185782-0-0] [lt=27][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:52.089924] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185784-0-0] [lt=23][errcode=-4201] hashtable not init [2025-01-08 00:50:52.094565] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8557, offset=2427187}, end_cursor:ObLogCursor{file_id=1, log_id=8558, offset=2427492}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946a4050, len:1741, log_data_len:305, seq:8557, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.095374] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3D-0-0] [lt=55] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:52.095427] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3D-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326251909938001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326251909938001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:52.098601] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:52.104051] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.110954] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326248101805, last_first_register_time_us=1736324449183670, register_time_us=1736326248101805, curr_time_us=1736326252110949) [2025-01-08 00:50:52.111059] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=44] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326252110999, last_submit_register_req_time_us=1736326252110999, register_parent_reason=1) [2025-01-08 00:50:52.111080] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:50:52.111172] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=55] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8558, offset=2427492}, end_cursor:ObLogCursor{file_id=1, log_id=8559, offset=2427789}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946a8050, len:1436, log_data_len:297, seq:8558, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.111635] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3D-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:52.111835] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3D-0-0] [lt=177] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326251909938001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326251909938001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:52.120592] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=101] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8559, offset=2427789}, end_cursor:ObLogCursor{file_id=1, log_id=8560, offset=2428086}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946ac050, len:1139, log_data_len:297, seq:8559, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.121556] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8560, offset=2428086}, end_cursor:ObLogCursor{file_id=1, log_id=8561, offset=2428383}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946b0050, len:842, log_data_len:297, seq:8560, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.121728] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=111] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17709, miss_cnt:0, cache_read_size:2166407, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:52.137767] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:52.148255] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:52.155472] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50896][T1002_L0_G2][T1002][YB42C0A89F8B-00062B2D1EABD780-0-0] [lt=94] get_min_unreplayed_log_info(lsn={lsn:2166407}, scn={val:1736326251703671004, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326252121666, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2166407}, next_to_submit_scn_:{val:1736326251703671004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2166285}, end_lsn:{lsn:2166407}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2166407}, readable_log_tail:{lsn:2166407}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64938361, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326251703671003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326251703671003, v:0}, data_checksum:2124447694, flag:40711}}, init_mode_version:0, accumulate_checksum:2484747650, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326251703671003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17709, miss_cnt:0, cache_read_size:2166407, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:52.161625] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58324][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D62C-0-0] [lt=203] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=1647, avg_log_submit_to_slide_time=20387) [2025-01-08 00:50:52.161772] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=90] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:52.164110] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] ====== checkpoint timer task ====== [2025-01-08 00:50:52.165156] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=101] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.165277] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=119] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.165462] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=54] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.165629] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=163] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:50:52.165652] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:52.165666] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326243478874001, v:0}) [2025-01-08 00:50:52.165673] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:50:52.165686] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=4] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:52.165726] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=34] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:52.165826] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:52.165872] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:52.165895] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:52.166795] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.166853] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=58] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.166900] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246900070001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.166963] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=61] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:50:52.166977] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:50:52.166991] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:52.167005] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:52.167021] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2143828}, scn={val:1736326251909938001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:52.169695] INFO [PALF] pread (log_reader.cpp:130) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1682) [2025-01-08 00:50:52.170610] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=27] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8561, offset=2428383}, end_cursor:ObLogCursor{file_id=1, log_id=8562, offset=2428680}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946b4050, len:545, log_data_len:297, seq:8561, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.170735] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326251909938001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:52.170766] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:52.170796] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=30] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:50:52.171641] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.171728] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=86] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.171772] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246984891000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.171886] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=111] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:50:52.171899] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:50:52.171913] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:52.171969] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=53] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:52.171988] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=16] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2176854}, scn={val:1736326251986852000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:52.172483] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:50:52.174459] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC36E-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:52.175762] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:328K/77:110648 disk:328K/77:110648 t1m2:44K/10:60619 t1m2g0:44K/10:60619 t1002m2:200K/49:120536 t1002m2g0:200K/49:120536 t1001m2:84K/18:111524 t1001m2g0:84K/18:111524 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:52.175830] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=56] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8562, offset=2428680}, end_cursor:ObLogCursor{file_id=1, log_id=8563, offset=2428977}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946b8050, len:4344, log_data_len:297, seq:8562, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.175972] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326251986852000, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:52.176005] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:52.176025] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:50:52.176054] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:52.177293] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.177388] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=94] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.177631] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=219] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246693924003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.177695] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=61] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:50:52.177707] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:50:52.177720] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:52.177734] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:52.177748] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2166529}, scn={val:1736326251703671004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:52.179600] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:52.179633] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=26][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:52.180317] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=22] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8563, offset=2428977}, end_cursor:ObLogCursor{file_id=1, log_id=8564, offset=2429274}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946c0050, len:4047, log_data_len:297, seq:8563, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.180570] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326251703671004, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:52.180655] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=78][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:52.180676] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:50:52.180686] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:50:52.199328] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AB9E-0-0] [lt=41] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.718940544128417969e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=385, Throttle Trigger(MB)=231, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.906445290688600380e+02) [2025-01-08 00:50:52.199964] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=162] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:52.202106] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] ====== traversal_flush timer task ====== [2025-01-08 00:50:52.202416] INFO [MDS] flush (mds_table_mgr.cpp:195) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=36] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326251909938002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.202619] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=166] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251909938001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.202735] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=84] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251986852000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.202855] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=87] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251703671004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.203368] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=509] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:50:52.204310] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.208213] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46099][TimerWK1][T1][Y0-0000000000000000-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1736326252204405000, start_id=1736326252204405000, end_id=1736326252204405001) [2025-01-08 00:50:52.211778] INFO [SHARE] delete_tenant_all (ob_disk_usage_table_operator.cpp:231) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=111] delete successful (ret=0, tenant_id=500) [2025-01-08 00:50:52.211899] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46099][TimerWK1][T1][Y0-0000000000000000-0-0] [lt=24] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:52.213514] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=15] get_stable_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:50:52.214895] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=13] get_stable_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:52.215024] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:50:52.216414] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [46099][TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-01-08 00:50:52.216431] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=15] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-01-08 00:50:52.216441] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-01-08 00:50:52.218033] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=7][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963659a18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27791} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae19970 trans_version={val:1736322640977792000, v:0} scn={val:1736322640977792000, v:0} tx_id={txid:1449} prev=0x2ad8dae197b8 next=(nil) modify_count=1 acc_checksum=460374033 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252216686000, v:0}, *this={this=0x2ad8dae19658 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad8dae19970 latest_compact_node=(nil) max_trans_version={val:1736326191777235000, v:0} max_trans_id=27369 max_elr_trans_version={val:1736326191777235000, v:0} max_elr_trans_id=27369 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1736322640977792000, v:0} min_modify_scn={val:1736322640977792000, v:0}}) [2025-01-08 00:50:52.218082] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=48][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963659a18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27791} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282117672, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282117672, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252216686000, v:0}, tx_id:{txid:27791}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27791}, tx_desc:{this:0x2ad8bcd27fa0, tx_id:{txid:27791}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252216686, active_ts:1736326252217943, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252315173, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.218168] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=85][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27791}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.218185] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27791} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.218214] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.218316] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=96][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c3b78540, local_datums:0x2ad8c3b78540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c3b783d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c3b783f3, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78403, hex: 6E417AF52D2B0600, int: 1736326252216686}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7840b, hex: 6E417AF52D2B0600, int: 1736326252216686}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78413, hex: 72D77F0100000000, int: 25155442}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7841b, hex: 00F0120200000000, int: 34795520}]}) [2025-01-08 00:50:52.218435] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:50) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=76] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=5510, insert_dup_release_count=5510, col_map_alloc_count=0, col_map_release_count=0) [2025-01-08 00:50:52.218467] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=18][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.218525] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=9] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27791} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.222827] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:52.222852] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:52.227393] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87019-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:52.227459] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:52.227474] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:52.227877] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] insert successful (ret=0, tenant_id=1, file_type=0, data_size=25155442) [2025-01-08 00:50:52.229312] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1335) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=26][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-01-08 00:50:52.229388] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=73][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.229402] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c21e0510, local_datums:0x2ad8c21e0510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e039c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c21e03a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad8c21e03c3, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03d5, hex: 206D7AF52D2B0600, int: 1736326252227872}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03dd, hex: 206D7AF52D2B0600, int: 1736326252227872}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03e5, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03ed, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:50:52.229512] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.229558] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27792} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.230462] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=27][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:52.232686] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] insert successful (ret=0, tenant_id=1, file_type=8, data_size=0) [2025-01-08 00:50:52.234125] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=52][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963659f18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27793} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3d9a8 trans_version={val:1736324383117027001, v:0} scn={val:1736324383117027001, v:0} tx_id={txid:14338} prev=(nil) next=(nil) modify_count=0 acc_checksum=1983279813 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252232880000, v:0}, *this={this=0x2ad8dae3d918 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3d9a8 latest_compact_node=(nil) max_trans_version={val:1736326191821630001, v:0} max_trans_id=27371 max_elr_trans_version={val:1736326191821630001, v:0} max_elr_trans_id=27371 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383117027001, v:0} min_modify_scn={val:1736324383117027001, v:0}}) [2025-01-08 00:50:52.234262] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=133][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963659f18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27793} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282133946, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282133946, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252232880000, v:0}, tx_id:{txid:27793}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27793}, tx_desc:{this:0x2ad8bcd36240, tx_id:{txid:27793}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252232880, active_ts:1736326252234059, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252335342, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.234351] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=89][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27793}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.234367] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27793} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.234454] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=85][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.234465] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c1d1a510, local_datums:0x2ad8c1d1a510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a39c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a3a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a3c3, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3d2, hex: B0807AF52D2B0600, int: 1736326252232880}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3da, hex: B0807AF52D2B0600, int: 1736326252232880}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3e2, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3ea, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:50:52.234584] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=89][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.234621] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27793} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.237368] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] insert successful (ret=0, tenant_id=1001, file_type=3, data_size=0) [2025-01-08 00:50:52.238614] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=24][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365a190 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27794} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3db90 trans_version={val:1736324383129258001, v:0} scn={val:1736324383129258001, v:0} tx_id={txid:14339} prev=(nil) next=(nil) modify_count=0 acc_checksum=450727420 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252238381000, v:0}, *this={this=0x2ad8dae3db00 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3db90 latest_compact_node=(nil) max_trans_version={val:1736326191826572001, v:0} max_trans_id=27372 max_elr_trans_version={val:1736326191826572001, v:0} max_elr_trans_id=27372 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383129258001, v:0} min_modify_scn={val:1736324383129258001, v:0}}) [2025-01-08 00:50:52.238654] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=40][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365a190 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27794} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282138395, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282138395, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252238381000, v:0}, tx_id:{txid:27794}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27794}, tx_desc:{this:0x2ad8bcd341e0, tx_id:{txid:27794}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252238381, active_ts:1736326252238381, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252340141, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.238724] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=68][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27794}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.238738] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=13][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27794} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.238761] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=22][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.238771] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c3b78510, local_datums:0x2ad8c3b78510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7839c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c3b783a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c3b783c3, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783d2, hex: 2D967AF52D2B0600, int: 1736326252238381}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783da, hex: 2D967AF52D2B0600, int: 1736326252238381}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783e2, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783ea, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:50:52.238845] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=45][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.238880] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=10] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27794} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.241631] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] insert successful (ret=0, tenant_id=1002, file_type=3, data_size=0) [2025-01-08 00:50:52.242809] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524F-0-0] [lt=25][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365a408 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27795} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae882a0 trans_version={val:1736326191857851001, v:0} scn={val:1736326191857851001, v:0} tx_id={txid:27373} prev=0x2ad9636501d8 next=(nil) modify_count=119 acc_checksum=1691663824 version=1736322640966574 type=1 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=3474012769566797877, snapshot_version={val:1736326252242117000, v:0}, *this={this=0x2ad8dae19d70 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2ad8dae882a0 latest_compact_node=0x2ad8dae882a0 max_trans_version={val:1736326191857851001, v:0} max_trans_id=27373 max_elr_trans_version={val:1736326191857851001, v:0} max_elr_trans_id=27373 latest_compact_ts=1736326191874037 last_compact_cnt=119 total_trans_node_cnt=120 max_modify_scn={val:1736326191857851001, v:0} min_modify_scn={val:1736322640988548000, v:0}}) [2025-01-08 00:50:52.242850] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524F-0-0] [lt=41][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365a408 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27795} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282142635, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282142635, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252242117000, v:0}, tx_id:{txid:27795}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27795}, tx_desc:{this:0x2ad8bcd382a0, tx_id:{txid:27795}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252242117, active_ts:1736326252242117, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252345113, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.242921] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524F-0-0] [lt=71][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27795}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.242936] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524F-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27795} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.242960] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524F-0-0] [lt=23][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.242969] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524F-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c21e0540, local_datums:0x2ad8c21e0540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c21e03d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c21e03f3, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0403, hex: C5A47AF52D2B0600, int: 1736326252242117}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e040b, hex: C5A47AF52D2B0600, int: 1736326252242117}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0413, hex: 026C040900000000, int: 151284738}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e041b, hex: 026C040900000000, int: 151284738}]}) [2025-01-08 00:50:52.243042] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524F-0-0] [lt=47][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.243078] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524F-0-0] [lt=10] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27795} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.246262] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] insert successful (ret=0, tenant_id=1, file_type=5, data_size=151284738) [2025-01-08 00:50:52.247558] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285252-0-0] [lt=24][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365a860 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27796} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3e2a0 trans_version={val:1736324383166524001, v:0} scn={val:1736324383166524001, v:0} tx_id={txid:14342} prev=(nil) next=(nil) modify_count=0 acc_checksum=2221000172 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252246923000, v:0}, *this={this=0x2ad8dae3e210 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3e2a0 latest_compact_node=(nil) max_trans_version={val:1736326191873223001, v:0} max_trans_id=27374 max_elr_trans_version={val:1736326191873223001, v:0} max_elr_trans_id=27374 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383166524001, v:0} min_modify_scn={val:1736324383166524001, v:0}}) [2025-01-08 00:50:52.247599] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285252-0-0] [lt=39][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365a860 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27796} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282147371, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282147371, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252246923000, v:0}, tx_id:{txid:27796}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27796}, tx_desc:{this:0x2ad8bcd3a300, tx_id:{txid:27796}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252246923, active_ts:1736326252246923, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252346173, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.247669] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285252-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27796}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.247683] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285252-0-0] [lt=13][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27796} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.247706] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285252-0-0] [lt=22][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.247716] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285252-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c1d1a540, local_datums:0x2ad8c1d1a540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3cc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c1d1a3f3, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a403, hex: 8BB77AF52D2B0600, int: 1736326252246923}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a40b, hex: 8BB77AF52D2B0600, int: 1736326252246923}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a413, hex: 710B080000000000, int: 527217}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a41b, hex: 00D03D0000000000, int: 4050944}]}) [2025-01-08 00:50:52.247791] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285252-0-0] [lt=46][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.247826] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285252-0-0] [lt=10] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27796} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.251020] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] insert successful (ret=0, tenant_id=1001, file_type=0, data_size=527217) [2025-01-08 00:50:52.252461] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285255-0-0] [lt=25][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365aae0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27797} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8da464ac0 trans_version={val:1736324505332247000, v:0} scn={val:1736324505332247000, v:0} tx_id={txid:15233} prev=0x2ad8da464908 next=(nil) modify_count=4 acc_checksum=3222268482 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252251922000, v:0}, *this={this=0x2ad8dae3e400 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8da464ac0 latest_compact_node=(nil) max_trans_version={val:1736326191902442001, v:0} max_trans_id=27375 max_elr_trans_version={val:1736326191902442001, v:0} max_elr_trans_id=27375 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324505332247000, v:0} min_modify_scn={val:1736324383199392001, v:0}}) [2025-01-08 00:50:52.252502] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285255-0-0] [lt=40][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365aae0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27797} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282152287, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282152287, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252251922000, v:0}, tx_id:{txid:27797}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27797}, tx_desc:{this:0x2ad8bcd3e3c0, tx_id:{txid:27797}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252251922, active_ts:1736326252251922, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252350808, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.252696] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285255-0-0] [lt=193][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27797}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.252712] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285255-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27797} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.252736] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285255-0-0] [lt=22][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.252746] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285255-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c3b78540, local_datums:0x2ad8c3b78540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783cc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c3b783d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c3b783f3, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78403, hex: 12CB7AF52D2B0600, int: 1736326252251922}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7840b, hex: 12CB7AF52D2B0600, int: 1736326252251922}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78413, hex: 1ED9080000000000, int: 579870}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7841b, hex: 00802F0000000000, int: 3112960}]}) [2025-01-08 00:50:52.252895] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285255-0-0] [lt=120][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.252932] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285255-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27797} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.253977] INFO [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:197) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] skip repeat study(ret=0, this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326192119682, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, current_timestamp=1736326252253975, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:50:52.255448] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=28] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325772835347000, snapshot_type=1, create_time=1736325772838939, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:50:52.255472] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325659399768000, snapshot_type=1, create_time=1736325659527637, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:50:52.255577] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=97] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325660825489000, snapshot_type=1, create_time=1736325660889571, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:50:52.255591] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325772082473004, snapshot_type=2, create_time=1736325772838939, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:50:52.255601] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325658409784004, snapshot_type=2, create_time=1736325659527637, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:50:52.255760] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=152] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325659910050004, snapshot_type=2, create_time=1736325660889571, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:50:52.255771] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325772583377000, snapshot_type=3, create_time=1736325772838939, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:50:52.255827] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=50] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325658978584000, snapshot_type=3, create_time=1736325659527637, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:50:52.255841] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325660410422000, snapshot_type=3, create_time=1736325660889571, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:50:52.255851] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325772838939, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:50:52.255860] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325659527637, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:50:52.255870] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325660889571, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:50:52.256059] INFO [MVCC] decide_reserved_snapshot_version_ (ob_multi_version_garbage_collector.cpp:620) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8] succeed to update global reserved snapshot(ret=0, this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326192119682, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, global_reserved_snapshot_={val:1736325658409784004, v:0}, reserved_snapshot={val:1736325658409784004, v:0}) [2025-01-08 00:50:52.256080] INFO [MVCC] refresh_ (ob_multi_version_garbage_collector.cpp:554) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] multi-version garbage collector refresh successfully(this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326192119682, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, collector={reserved_snapshot_version:{val:1736325658409784004, v:0}, reserved_snapshot_type:2, reserved_status:-1, reserved_create_time:1736325659527637, reserved_addr:"192.168.159.138:2882", is_this_server_disabled:false, status:0}) [2025-01-08 00:50:52.256097] INFO [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:232) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17] repeat refresh successfully(ret=0, this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326252256096, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, current_timestamp=1736326252254005, GARBAGE_COLLECT_RETRY_INTERVAL=60000000) [2025-01-08 00:50:52.256109] INFO [MVCC] repeat_reclaim (ob_multi_version_garbage_collector.cpp:267) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12] skip repeat reclaim(ret=0, this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326252256096, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, current_timestamp=1736326252256108, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:50:52.256140] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=41] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:52.264942] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB4E-0-0] [lt=22][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193757178778, cache_obj->added_lc()=false, cache_obj->get_object_id()=13555, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:52.267296] INFO [STORAGE.COMPACTION] schedule_all_tablets_minor (ob_tenant_tablet_scheduler.cpp:575) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854AE-0-0] [lt=48] start schedule all tablet minor merge(minor_ls_tablet_iter_={ls_idx:-1, ls_ids:[{id:1}, {id:1001}, {id:1002}, {id:1003}], tablet_ids:{tablet_idx:0, tablet_cnt:0, array:[], is_inited:false}, schedule_tablet_cnt:0, max_batch_tablet_cnt:50000}) [2025-01-08 00:50:52.268423] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326252257312000, start_id=1736326252257312000, end_id=1736326252257312001) [2025-01-08 00:50:52.268570] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:52.277512] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=37] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:52.278925] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] clock out of order(cur_ts=1736326252278922, cur_ts_=1736326252257312, delta=21610) [2025-01-08 00:50:52.279602] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A441F-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=21, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:52.279628] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A441F-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=15147, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:52.279639] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A441F-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=11, avg_log_submit_to_flush_time=1521, avg_log_submit_to_slide_time=15136) [2025-01-08 00:50:52.279652] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A441F-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17885, log_task={header:{begin_lsn:{lsn:2176854}, end_lsn:{lsn:2176976}, log_id:17885, min_scn:{val:1736326251986852001, v:0}, max_scn:{val:1736326251986852001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2176732}, prev_proposal_id:1, committed_end_lsn:{lsn:2176854}, data_checksum:1715074492, accum_checksum:2763810404, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326252174804, freeze_ts:1736326252174804, submit_ts:1736326252174814, flushed_ts:1736326252178061, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:3247}, fs_cb_begin_ts=1736326252279591, log_life_time=104787) [2025-01-08 00:50:52.279846] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] insert successful (ret=0, tenant_id=1002, file_type=0, data_size=579870) [2025-01-08 00:50:52.281395] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285258-0-0] [lt=21][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365ad68 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27799} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:6206} write_epoch=1, prev=this=0x2ad8dae3e680 trans_version={val:1736324383216166001, v:0} scn={val:1736324383216166001, v:0} tx_id={txid:14344} prev=(nil) next=(nil) modify_count=0 acc_checksum=3114694747 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252257312001, v:0}, *this={this=0x2ad8dae3e5f0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3e680 latest_compact_node=(nil) max_trans_version={val:1736326191913018001, v:0} max_trans_id=27376 max_elr_trans_version={val:1736326191913018001, v:0} max_elr_trans_id=27376 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383216166001, v:0} min_modify_scn={val:1736324383216166001, v:0}}) [2025-01-08 00:50:52.281441] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285258-0-0] [lt=46][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365ad68 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27799} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:6206} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282181042, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282181042, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252257312001, v:0}, tx_id:{txid:27799}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27799}, tx_desc:{this:0x2ad8bcd3c360, tx_id:{txid:27799}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252257312, active_ts:1736326252257312, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252376430, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:6206}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.281525] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285258-0-0] [lt=83][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27799}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:6206}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.281542] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285258-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27799} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.281571] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285258-0-0] [lt=26][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.281532] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:50:52.281581] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285258-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c21e0510, local_datums:0x2ad8c21e0510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e039c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c21e03a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad8c21e03c3, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03d5, hex: 20E07AF52D2B0600, int: 1736326252257312}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03dd, hex: 20E07AF52D2B0600, int: 1736326252257312}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03e5, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03ed, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:50:52.281715] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285258-0-0] [lt=53][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.281768] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285258-0-0] [lt=14] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6207}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27799} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.282071] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=113] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1454, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:52.283703] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9A-0-0] [lt=43] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:52.283734] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9A-0-0] [lt=29] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:52.295495] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540F-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:52.299264] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=30] success to update min reserved snapshot(reserved_snapshot=1736324445675194000, duration=1800, snapshot_gc_ts=1736326245675194000) [2025-01-08 00:50:52.299326] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10815) [2025-01-08 00:50:52.304420] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.314749] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=12] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:52.314864] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=27] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:52.315324] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=220][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18642647}], scn_array=[cnt:1, {val:1736326251909938003, v:0}], curr_size=122, accum_size=488, time_cost=61849) [2025-01-08 00:50:52.317095] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84171914}], scn_array=[cnt:1, {val:1736326252257312000, v:0}], curr_size=586, accum_size=8890, time_cost=48167) [2025-01-08 00:50:52.317290] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=10341, avg_group_log_size=574) [2025-01-08 00:50:52.319679] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.319718] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.321689] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] insert successful (ret=0, tenant_id=1001, file_type=8, data_size=0) [2025-01-08 00:50:52.323449] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285259-0-0] [lt=27][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365aff8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27800} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3e878 trans_version={val:1736324383240856001, v:0} scn={val:1736324383240856001, v:0} tx_id={txid:14345} prev=(nil) next=(nil) modify_count=0 acc_checksum=3482093884 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252313883000, v:0}, *this={this=0x2ad8dae3e7e8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3e878 latest_compact_node=(nil) max_trans_version={val:1736326191952504001, v:0} max_trans_id=27378 max_elr_trans_version={val:1736326191952504001, v:0} max_elr_trans_id=27378 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383240856001, v:0} min_modify_scn={val:1736324383240856001, v:0}}) [2025-01-08 00:50:52.323495] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285259-0-0] [lt=46][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365aff8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27800} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282222960, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282222960, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252313883000, v:0}, tx_id:{txid:27800}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27800}, tx_desc:{this:0x2ad8bcd46540, tx_id:{txid:27800}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252313883, active_ts:1736326252313883, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252421939, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.323580] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285259-0-0] [lt=83][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27800}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.323596] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285259-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27800} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.323626] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285259-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.323636] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285259-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c1d1a510, local_datums:0x2ad8c1d1a510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a39c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a3a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad8c1d1a3c3, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3d5, hex: 1BBD7BF52D2B0600, int: 1736326252313883}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3dd, hex: 1BBD7BF52D2B0600, int: 1736326252313883}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3e5, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3ed, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:50:52.323732] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285259-0-0] [lt=54][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.323785] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285259-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27800} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.325151] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2176976}], scn_array=[cnt:1, {val:1736326251986852002, v:0}], curr_size=122, accum_size=732, time_cost=45014) [2025-01-08 00:50:52.325291] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=45] update reserved snapshot finished(cost_ts=26030, reserved_snapshot=1736324445675194000) [2025-01-08 00:50:52.327977] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] insert successful (ret=0, tenant_id=1002, file_type=8, data_size=0) [2025-01-08 00:50:52.331869] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] insert successful (ret=0, tenant_id=500, file_type=5, data_size=872415232) [2025-01-08 00:50:52.333082] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728525F-0-0] [lt=25][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365b450 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27802} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad963651220 trans_version={val:1736326192005923001, v:0} scn={val:1736326192005923001, v:0} tx_id={txid:27380} prev=0x2ad8dae88440 next=(nil) modify_count=60 acc_checksum=2782402695 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252325058002, v:0}, *this={this=0x2ad8dae3ec78 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad963651220 latest_compact_node=0x2ad8dae88440 max_trans_version={val:1736326192005923001, v:0} max_trans_id=27380 max_elr_trans_version={val:1736326192005923001, v:0} max_elr_trans_id=27380 latest_compact_ts=1736326192026746 last_compact_cnt=59 total_trans_node_cnt=61 max_modify_scn={val:1736326192005923001, v:0} min_modify_scn={val:1736324383282537001, v:0}}) [2025-01-08 00:50:52.333125] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728525F-0-0] [lt=42][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365b450 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27802} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282232910, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282232910, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252325058002, v:0}, tx_id:{txid:27802}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27802}, tx_desc:{this:0x2ad8bcd40420, tx_id:{txid:27802}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252325058, active_ts:1736326252325058, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252432513, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.333341] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728525F-0-0] [lt=214][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27802}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.333371] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728525F-0-0] [lt=28][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27802} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.333399] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728525F-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.333410] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728525F-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c21e0570, local_datums:0x2ad8c21e0570}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03fc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c21e0404, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0413, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e041b, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c21e0423, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0433, hex: C2E87BF52D2B0600, int: 1736326252325058}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e043b, hex: C2E87BF52D2B0600, int: 1736326252325058}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0443, hex: A989900600000000, int: 110135721}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e044b, hex: A989900600000000, int: 110135721}]}) [2025-01-08 00:50:52.333494] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728525F-0-0] [lt=51][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.333535] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728525F-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27802} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.336454] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=25] insert successful (ret=0, tenant_id=1001, file_type=5, data_size=110135721) [2025-01-08 00:50:52.337723] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285262-0-0] [lt=24][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365b8a8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27803} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad963651678 trans_version={val:1736326192028912001, v:0} scn={val:1736326192028912001, v:0} tx_id={txid:27381} prev=0x2ad8dae884f8 next=(nil) modify_count=60 acc_checksum=3601012833 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252335436000, v:0}, *this={this=0x2ad8dae3ee68 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad963651678 latest_compact_node=0x2ad8dae884f8 max_trans_version={val:1736326192028912001, v:0} max_trans_id=27381 max_elr_trans_version={val:1736326192028912001, v:0} max_elr_trans_id=27381 latest_compact_ts=1736326192046887 last_compact_cnt=59 total_trans_node_cnt=61 max_modify_scn={val:1736326192028912001, v:0} min_modify_scn={val:1736324383301306001, v:0}}) [2025-01-08 00:50:52.337763] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285262-0-0] [lt=39][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365b8a8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27803} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282237552, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282237552, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252335436000, v:0}, tx_id:{txid:27803}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27803}, tx_desc:{this:0x2ad8bcd444e0, tx_id:{txid:27803}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252335436, active_ts:1736326252335436, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252437075, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.337834] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285262-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27803}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.337848] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285262-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27803} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.337871] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285262-0-0] [lt=22][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.337881] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285262-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c3b78570, local_datums:0x2ad8c3b78570}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783fc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c3b78404, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78413, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7841b, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c3b78423, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78433, hex: 4C117CF52D2B0600, int: 1736326252335436}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7843b, hex: 4C117CF52D2B0600, int: 1736326252335436}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78443, hex: FCB37F1100000000, int: 293581820}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7844b, hex: FCB37F1100000000, int: 293581820}]}) [2025-01-08 00:50:52.338010] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285262-0-0] [lt=100][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.338047] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285262-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27803} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.339260] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:52.339470] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2166529}], scn_array=[cnt:1, {val:1736326252202687000, v:0}], curr_size=122, accum_size=366, time_cost=82806) [2025-01-08 00:50:52.343672] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] insert successful (ret=0, tenant_id=1002, file_type=5, data_size=293581820) [2025-01-08 00:50:52.345054] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285265-0-0] [lt=25][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365bd00 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27804} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae1a6d0 trans_version={val:1736322640997837000, v:0} scn={val:1736322640997837000, v:0} tx_id={txid:1453} prev=0x2ad8dae1a518 next=(nil) modify_count=1 acc_checksum=311300284 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:8} write_epoch=3, snapshot_version={val:1736326252343153000, v:0}, *this={this=0x2ad8dae1a3b8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad8dae1a6d0 latest_compact_node=(nil) max_trans_version={val:1736326192046973001, v:0} max_trans_id=27382 max_elr_trans_version={val:1736326192046973001, v:0} max_elr_trans_id=27382 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1736322640997837000, v:0} min_modify_scn={val:1736322640997837000, v:0}}) [2025-01-08 00:50:52.345095] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285265-0-0] [lt=40][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365bd00 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27804} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282244807, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282244807, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252343153000, v:0}, tx_id:{txid:27804}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27804}, tx_desc:{this:0x2ad8bcd485a0, tx_id:{txid:27804}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252343153, active_ts:1736326252343153, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252445409, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.345169] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285265-0-0] [lt=73][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27804}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.345255] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285265-0-0] [lt=84][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27804} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.345282] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285265-0-0] [lt=25][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.345291] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285265-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c1d1a540, local_datums:0x2ad8c1d1a540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad8c1d1a3f3, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a404, hex: 712F7CF52D2B0600, int: 1736326252343153}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a40c, hex: 712F7CF52D2B0600, int: 1736326252343153}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a414, hex: 00C0E20200000000, int: 48414720}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a41c, hex: 00C0E20200000000, int: 48414720}]}) [2025-01-08 00:50:52.345370] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285265-0-0] [lt=48][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.345408] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285265-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27804} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.354544] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] insert successful (ret=0, tenant_id=1, file_type=7, data_size=48414720) [2025-01-08 00:50:52.355597] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:50:52.355618] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] try gc retain ctx [2025-01-08 00:50:52.355626] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-01-08 00:50:52.355655] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326251909938003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326241890747004, v:0}, keep_alive_scn_:{val:1736326246900070004, v:0}, update_ts_:1736326252354103}) [2025-01-08 00:50:52.356000] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285268-0-0] [lt=69][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365bf80 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27805} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae1ab48 trans_version={val:1736322641000286001, v:0} scn={val:1736322641000286001, v:0} tx_id={txid:1454} prev=0x2ad8dae1a990 next=(nil) modify_count=1 acc_checksum=1712141431 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252354103000, v:0}, *this={this=0x2ad8dae1a830 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad8dae1ab48 latest_compact_node=(nil) max_trans_version={val:1736326192082540000, v:0} max_trans_id=27383 max_elr_trans_version={val:1736326192082540000, v:0} max_elr_trans_id=27383 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1736322641000286001, v:0} min_modify_scn={val:1736322641000286001, v:0}}) [2025-01-08 00:50:52.356123] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285268-0-0] [lt=121][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365bf80 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27805} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282255716, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282255716, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252354103000, v:0}, tx_id:{txid:27805}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27805}, tx_desc:{this:0x2ad8bcd4c660, tx_id:{txid:27805}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252354103, active_ts:1736326252354103, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252448907, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.356297] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285268-0-0] [lt=172][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27805}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.356317] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285268-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27805} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.356345] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285268-0-0] [lt=26][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.356407] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285268-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c21e0540, local_datums:0x2ad8c21e0540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c21e03d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c21e03f3, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0403, hex: 375A7CF52D2B0600, int: 1736326252354103}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e040b, hex: 375A7CF52D2B0600, int: 1736326252354103}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0413, hex: 0060020000000000, int: 155648}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e041b, hex: 0060020000000000, int: 155648}]}) [2025-01-08 00:50:52.356552] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285268-0-0] [lt=98][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.356599] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285268-0-0] [lt=13] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27805} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.356561] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:52.356688] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=86, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:52.356745] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] finish update min start scn(max_decided_scn={val:1736326251909938003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326241890747004, v:0}, keep_alive_scn_:{val:1736326246900070004, v:0}, update_ts_:1736326252354103}) [2025-01-08 00:50:52.357680] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251909938001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:52.357833] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=101] finish update min start scn(max_decided_scn={val:1736326251986852001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326244981816000, v:0}, keep_alive_scn_:{val:1736326249981471000, v:0}, update_ts_:1736326252354103}) [2025-01-08 00:50:52.358653] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251986852000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:52.358825] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=160] finish update min start scn(max_decided_scn={val:1736326251703671004, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326243184127002, v:0}, keep_alive_scn_:{val:1736326248201055002, v:0}, update_ts_:1736326252354103}) [2025-01-08 00:50:52.359536] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251703671004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:52.359642] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:50:52.361725] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36E5-0-0] [lt=26][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17800, log_task={header:{begin_lsn:{lsn:2166529}, end_lsn:{lsn:2166651}, log_id:17800, min_scn:{val:1736326252202687000, v:0}, max_scn:{val:1736326252202687000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2166407}, prev_proposal_id:1, committed_end_lsn:{lsn:2166529}, data_checksum:940009244, accum_checksum:3411007118, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326252256374, freeze_ts:1736326252256374, submit_ts:1736326252256381, flushed_ts:1736326252339537, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:83156}, fs_cb_begin_ts=1736326252361717, log_life_time=105343) [2025-01-08 00:50:52.362920] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=25] insert successful (ret=0, tenant_id=1, file_type=4, data_size=155648) [2025-01-08 00:50:52.364535] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526B-0-0] [lt=38][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365c200 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27806} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3f5e0 trans_version={val:1736324383421133001, v:0} scn={val:1736324383421133001, v:0} tx_id={txid:14352} prev=(nil) next=(nil) modify_count=0 acc_checksum=2494219223 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252363406000, v:0}, *this={this=0x2ad8dae3f550 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3f5e0 latest_compact_node=(nil) max_trans_version={val:1736326192083806001, v:0} max_trans_id=27384 max_elr_trans_version={val:1736326192083806001, v:0} max_elr_trans_id=27384 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383421133001, v:0} min_modify_scn={val:1736324383421133001, v:0}}) [2025-01-08 00:50:52.364600] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526B-0-0] [lt=63][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365c200 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27806} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282264251, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282264251, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252363406000, v:0}, tx_id:{txid:27806}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27806}, tx_desc:{this:0x2ad8bcd4a600, tx_id:{txid:27806}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252363406, active_ts:1736326252363406, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252464203, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.364680] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526B-0-0] [lt=79][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27806}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.364694] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526B-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27806} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.364720] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526B-0-0] [lt=24][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.364729] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:81) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854AE-0-0] [lt=50] MergeLoopTask(cost_ts=97865) [2025-01-08 00:50:52.364730] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526B-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c3b78540, local_datums:0x2ad8c3b78540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783cc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c3b783d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad8c3b783f3, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78404, hex: 8E7E7CF52D2B0600, int: 1736326252363406}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7840c, hex: 8E7E7CF52D2B0600, int: 1736326252363406}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78414, hex: 00D0F70000000000, int: 16240640}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7841c, hex: 00D0F70000000000, int: 16240640}]}) [2025-01-08 00:50:52.364869] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526B-0-0] [lt=79][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.364913] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526B-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:1293}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27806} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.371037] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=21] insert successful (ret=0, tenant_id=1001, file_type=7, data_size=16240640) [2025-01-08 00:50:52.372501] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526E-0-0] [lt=25][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365c480 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27807} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8da466638 trans_version={val:1736324505384273001, v:0} scn={val:1736324505384273001, v:0} tx_id={txid:15242} prev=0x2ad8da466480 next=(nil) modify_count=4 acc_checksum=590613958 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252371763000, v:0}, *this={this=0x2ad8dae3f740 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8da466638 latest_compact_node=(nil) max_trans_version={val:1736326192097973000, v:0} max_trans_id=27385 max_elr_trans_version={val:1736326192097973000, v:0} max_elr_trans_id=27385 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324505384273001, v:0} min_modify_scn={val:1736324383455288001, v:0}}) [2025-01-08 00:50:52.372565] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526E-0-0] [lt=64][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365c480 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27807} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282272158, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282272158, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252371763000, v:0}, tx_id:{txid:27807}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27807}, tx_desc:{this:0x2ad8bcd52780, tx_id:{txid:27807}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252371763, active_ts:1736326252371763, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252470107, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.372713] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526E-0-0] [lt=146][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27807}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.372733] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526E-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27807} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.372758] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526E-0-0] [lt=23][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.372767] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526E-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c1d1a540, local_datums:0x2ad8c1d1a540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3cc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad8c1d1a3f3, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a404, hex: 339F7CF52D2B0600, int: 1736326252371763}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a40c, hex: 339F7CF52D2B0600, int: 1736326252371763}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a414, hex: 00E0B60000000000, int: 11984896}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a41c, hex: 00E0B60000000000, int: 11984896}]}) [2025-01-08 00:50:52.372920] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526E-0-0] [lt=72][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.372963] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728526E-0-0] [lt=14] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:3594}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27807} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.373787] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:50:52.374611] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:50:52.374564] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58349][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:52.376824] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58349][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6743-0-0] [lt=178] init point thread id with(&point=0x5572d50e2480, idx_=1005, point=[thread id=58349, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58349) [2025-01-08 00:50:52.377230] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=50] insert successful (ret=0, tenant_id=1002, file_type=7, data_size=11984896) [2025-01-08 00:50:52.381304] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=35] insert successful (ret=0, tenant_id=500, file_type=4, data_size=45056) [2025-01-08 00:50:52.382492] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285274-0-0] [lt=39][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365c8d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27809} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae1b190 trans_version={val:1736322641036856001, v:0} scn={val:1736322641036856001, v:0} tx_id={txid:1457} prev=0x2ad8dae1afd8 next=(nil) modify_count=1 acc_checksum=3703409285 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252381290000, v:0}, *this={this=0x2ad8dae1ae78 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad8dae1b190 latest_compact_node=(nil) max_trans_version={val:1736326192108026001, v:0} max_trans_id=27387 max_elr_trans_version={val:1736326192108026001, v:0} max_elr_trans_id=27387 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1736322641036856001, v:0} min_modify_scn={val:1736322641036856001, v:0}}) [2025-01-08 00:50:52.382557] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285274-0-0] [lt=63][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365c8d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27809} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282282305, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282282305, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252381290000, v:0}, tx_id:{txid:27809}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27809}, tx_desc:{this:0x2ad8bcd588a0, tx_id:{txid:27809}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252381290, active_ts:1736326252381290, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252481699, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.382637] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285274-0-0] [lt=79][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27809}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.382651] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285274-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27809} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.382677] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285274-0-0] [lt=24][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.382687] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285274-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c3b78540, local_datums:0x2ad8c3b78540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c3b783d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c3b783f3, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78403, hex: 6AC47CF52D2B0600, int: 1736326252381290}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7840b, hex: 6AC47CF52D2B0600, int: 1736326252381290}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78413, hex: 00D0CF0000000000, int: 13619200}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7841b, hex: 00D0CF0000000000, int: 13619200}]}) [2025-01-08 00:50:52.382771] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285274-0-0] [lt=50][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.382854] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285274-0-0] [lt=54] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27809} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.384011] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=28] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:52.385144] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=1451, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:52.385226] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=80] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10185) [2025-01-08 00:50:52.385247] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=455, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:52.385281] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=101] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=14, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:52.386662] INFO [LIB] stat (utility.h:1210) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA380B-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=10, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:52.386698] INFO [LIB] stat (utility.h:1210) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA380B-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4101, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:52.386711] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA380B-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=96, avg_log_submit_to_flush_time=1980, avg_log_submit_to_slide_time=4004) [2025-01-08 00:50:52.386899] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10185, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:52.387371] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326251703671003, v:0}, new_max_version={val:1736326251703671003, v:0}, last_min_version={val:1736326251487377004, v:0}, last_max_version={val:1736326251487377004, v:0}, total_time_us=5570, delta_us=683698) [2025-01-08 00:50:52.387452] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=76] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326251703671003, v:0}, min_version={val:1736326251703671003, v:0}, max_version={val:1736326251703671003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:52.388086] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] insert successful (ret=0, tenant_id=1, file_type=1, data_size=13619200) [2025-01-08 00:50:52.389486] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285277-0-0] [lt=76][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365cb50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27810} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3fed8 trans_version={val:1736324383590655001, v:0} scn={val:1736324383590655001, v:0} tx_id={txid:14358} prev=(nil) next=(nil) modify_count=0 acc_checksum=1106164579 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252388861000, v:0}, *this={this=0x2ad8dae3fe48 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3fed8 latest_compact_node=(nil) max_trans_version={val:1736326192120341002, v:0} max_trans_id=27389 max_elr_trans_version={val:1736326192120341002, v:0} max_elr_trans_id=27389 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383590655001, v:0} min_modify_scn={val:1736324383590655001, v:0}}) [2025-01-08 00:50:52.389558] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285277-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365cb50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27810} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282289303, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282289303, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252388861000, v:0}, tx_id:{txid:27810}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27810}, tx_desc:{this:0x2ad8bcd56840, tx_id:{txid:27810}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252388861, active_ts:1736326252388861, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252490204, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.389660] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285277-0-0] [lt=101][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27810}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.389679] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285277-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27810} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.389704] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285277-0-0] [lt=22][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.389712] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285277-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c21e0540, local_datums:0x2ad8c21e0540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03cc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c21e03d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c21e03f3, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0403, hex: FDE17CF52D2B0600, int: 1736326252388861}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e040b, hex: FDE17CF52D2B0600, int: 1736326252388861}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0413, hex: 00E0010000000000, int: 122880}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e041b, hex: 00E0010000000000, int: 122880}]}) [2025-01-08 00:50:52.389875] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285277-0-0] [lt=90][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.389943] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285277-0-0] [lt=39] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27810} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.391109] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:52.391244] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=127] temporary file module use no blocks [2025-01-08 00:50:52.391255] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:52.391267] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:52.391279] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193756304344, flush_io_finished_ret_:0, flush_io_finished_round_:1911, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:52.394510] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=73] insert successful (ret=0, tenant_id=1001, file_type=4, data_size=122880) [2025-01-08 00:50:52.395876] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728527A-0-0] [lt=71][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365cdd0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27811} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad963652ba0 trans_version={val:1736326192135668001, v:0} scn={val:1736326192135668001, v:0} tx_id={txid:27390} prev=0x2ad8dae88680 next=(nil) modify_count=60 acc_checksum=1258229203 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252395768000, v:0}, *this={this=0x2ad8dae40038 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad963652ba0 latest_compact_node=0x2ad8dae88680 max_trans_version={val:1736326192135668001, v:0} max_trans_id=27390 max_elr_trans_version={val:1736326192135668001, v:0} max_elr_trans_id=27390 latest_compact_ts=1736326192150383 last_compact_cnt=59 total_trans_node_cnt=61 max_modify_scn={val:1736326192135668001, v:0} min_modify_scn={val:1736324383643182001, v:0}}) [2025-01-08 00:50:52.395940] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728527A-0-0] [lt=62][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365cdd0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27811} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282295702, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282295702, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252395768000, v:0}, tx_id:{txid:27811}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27811}, tx_desc:{this:0x2ad8bcd4e6c0, tx_id:{txid:27811}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252395768, active_ts:1736326252395768, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252494697, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.396014] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728527A-0-0] [lt=74][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27811}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.396029] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728527A-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27811} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.396052] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728527A-0-0] [lt=22][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.396062] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728527A-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c1d1a540, local_datums:0x2ad8c1d1a540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3cc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c1d1a3f3, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a403, hex: AFF87CF52D2B0600, int: 1736326252394671}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a40b, hex: AFF87CF52D2B0600, int: 1736326252394671}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a413, hex: 0020250000000000, int: 2433024}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a41b, hex: 0020250000000000, int: 2433024}]}) [2025-01-08 00:50:52.396157] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728527A-0-0] [lt=68][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.396256] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728527A-0-0] [lt=13] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27811} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.397002] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:52.397047] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=40] temporary file module use no blocks [2025-01-08 00:50:52.397058] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:52.397070] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:52.397081] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193756309181, flush_io_finished_ret_:0, flush_io_finished_round_:3739, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:52.398022] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1736326252181801000, v:0}, base_version={val:1736326251795670000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326252196703000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326252398004, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326252196703000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326252479154, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326252181801000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326252293249, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:50:52.402867] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=57] insert successful (ret=0, tenant_id=1002, file_type=4, data_size=2433024) [2025-01-08 00:50:52.404973] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=125][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.408342] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=40] insert successful (ret=0, tenant_id=500, file_type=1, data_size=0) [2025-01-08 00:50:52.409644] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285280-0-0] [lt=25][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365d3f0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27813} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae40540 trans_version={val:1736324383691496001, v:0} scn={val:1736324383691496001, v:0} tx_id={txid:14361} prev=(nil) next=(nil) modify_count=0 acc_checksum=1965065760 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252408830000, v:0}, *this={this=0x2ad8dae404b0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae40540 latest_compact_node=(nil) max_trans_version={val:1736326192163831000, v:0} max_trans_id=27392 max_elr_trans_version={val:1736326192163831000, v:0} max_elr_trans_id=27392 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383691496001, v:0} min_modify_scn={val:1736324383691496001, v:0}}) [2025-01-08 00:50:52.409689] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285280-0-0] [lt=43][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365d3f0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27813} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282309456, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282309456, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252408830000, v:0}, tx_id:{txid:27813}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27813}, tx_desc:{this:0x2ad8bcd5c960, tx_id:{txid:27813}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252408830, active_ts:1736326252408830, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252511617, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.409769] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285280-0-0] [lt=79][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27813}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.409784] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285280-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27813} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.409811] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285280-0-0] [lt=24][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.409820] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285280-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c21e0540, local_datums:0x2ad8c21e0540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03cc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c21e03d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c21e03f3, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0403, hex: FE2F7DF52D2B0600, int: 1736326252408830}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e040b, hex: FE2F7DF52D2B0600, int: 1736326252408830}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e0413, hex: 0000BA0000000000, int: 12189696}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e041b, hex: 0000BA0000000000, int: 12189696}]}) [2025-01-08 00:50:52.409909] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285280-0-0] [lt=51][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.409952] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285280-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27813} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.410812] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15] ts response task statistics(alloc_count=59454, free_count=29726) [2025-01-08 00:50:52.412585] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=32] ====== traversal_flush timer task ====== [2025-01-08 00:50:52.412717] INFO [MDS] flush (mds_table_mgr.cpp:195) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EC5-0-0] [lt=21] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326252398361000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259284252000, latest_log_ts:{val:1736326242623339001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.412859] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EC5-0-0] [lt=136] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:50:52.413858] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] insert successful (ret=0, tenant_id=1001, file_type=1, data_size=12189696) [2025-01-08 00:50:52.415074] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285283-0-0] [lt=24] need check schema(new_schema_version=1736326180221088, cached_tenant_schema_version=1736326180221088, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=INSERT INTO __all_space_usage (tenant_id, svr_ip, svr_port, start_seq, file_type, data_size, used_size) VALUES (?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE data_size = ?, used_size = ?) [2025-01-08 00:50:52.415300] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285283-0-0] [lt=21][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365d670 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27814} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8da467a00 trans_version={val:1736324505450324000, v:0} scn={val:1736324505450324000, v:0} tx_id={txid:15249} prev=0x2ad8da467848 next=(nil) modify_count=4 acc_checksum=3125179850 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252414747000, v:0}, *this={this=0x2ad8dae406a0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8da467a00 latest_compact_node=(nil) max_trans_version={val:1736326192182157001, v:0} max_trans_id=27393 max_elr_trans_version={val:1736326192182157001, v:0} max_elr_trans_id=27393 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324505450324000, v:0} min_modify_scn={val:1736324383710480001, v:0}}) [2025-01-08 00:50:52.415340] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285283-0-0] [lt=40][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365d670 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27814} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282315035, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282315035, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252414747000, v:0}, tx_id:{txid:27814}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27814}, tx_desc:{this:0x2ad8bcd62a80, tx_id:{txid:27814}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252414747, active_ts:1736326252414747, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252516204, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.415412] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285283-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27814}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.415472] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285283-0-0] [lt=59][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27814} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.415498] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285283-0-0] [lt=24][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.415507] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285283-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c3b78540, local_datums:0x2ad8c3b78540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783cc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c3b783d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c3b783f3, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78403, hex: 1B477DF52D2B0600, int: 1736326252414747}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7840b, hex: 1B477DF52D2B0600, int: 1736326252414747}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78413, hex: 0060870000000000, int: 8871936}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7841b, hex: 0060870000000000, int: 8871936}]}) [2025-01-08 00:50:52.415584] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285283-0-0] [lt=48][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.415620] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285283-0-0] [lt=10] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27814} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.418295] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] insert successful (ret=0, tenant_id=1002, file_type=1, data_size=8871936) [2025-01-08 00:50:52.419632] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1335) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285286-0-0] [lt=23][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-01-08 00:50:52.419651] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285286-0-0] [lt=18][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.419663] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285286-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c1d1a4e0, local_datums:0x2ad8c1d1a4e0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a36c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a374, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a383, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a38b, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a393, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3a2, hex: 6C557DF52D2B0600, int: 1736326252418412}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3aa, hex: 6C557DF52D2B0600, int: 1736326252418412}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3b2, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3ba, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:50:52.419756] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285286-0-0] [lt=48][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.419820] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285286-0-0] [lt=39] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27815} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.421291] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=37] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:50:52.421321] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=27] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:50:52.421334] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:50:52.421344] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:52.424351] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=130, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}]) [2025-01-08 00:50:52.424700] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=345] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:52.424733] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] insert successful (ret=0, tenant_id=1, file_type=3, data_size=0) [2025-01-08 00:50:52.430704] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E6-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:52.430821] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E6-0-0] [lt=99] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326252202687000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326252202687000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:52.445966] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=21] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:52.455055] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:52.456857] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:52.460648] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.460685] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.461173] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=66] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=658198528, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=415543296, wash_ratio=6, tenant_id=1001) [2025-01-08 00:50:52.461596] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=387] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:50:52.461619] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=20] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2795, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:52.461962] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=149] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1116487680, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2104737792, wash_ratio=6, tenant_id=1002) [2025-01-08 00:50:52.462049] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=54] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:50:52.462067] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=17] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:52.467269] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] no need to update min/max version(new_version={val:1736326252067809000, v:0}, min_version={val:1736326252067809000, v:0}, max_version={val:1736326252067809000, v:0}) [2025-01-08 00:50:52.504959] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.506372] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1736326252506368, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007443, add_bytes: 0 [2025-01-08 00:50:52.519220] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.539687] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:52.542368] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.542540] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:52.542632] INFO pn_ratelimit (group.c:653) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=37] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:50:52.544207] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.545930] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58307][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=9] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:50:52.556364] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:52.556473] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=89] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:50:52.556496] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:50:52.556510] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:52.558029] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=62] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:50:52.559071] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F8E-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:52.559097] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F8E-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6451, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:52.559939] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=31] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:50:52.561347] INFO [LIB] stat (utility.h:1210) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F8F-0-0] [lt=90] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:52.561382] INFO [LIB] stat (utility.h:1210) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F8F-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11782, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:52.561454] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F8F-0-0] [lt=70] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=15, avg_log_gen_to_submit_time=123, avg_log_submit_to_flush_time=9496, avg_log_submit_to_slide_time=11659) [2025-01-08 00:50:52.561521] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=278] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1634, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:52.563629] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCE-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:52.563723] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCE-0-0] [lt=86] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=97) [2025-01-08 00:50:52.572953] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:52.573284] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36E9-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:52.573331] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36E9-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:52.574096] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:50:52.574451] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4425-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:52.581472] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=35] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:52.583894] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=10651, avg_group_log_size=591) [2025-01-08 00:50:52.583935] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10185) [2025-01-08 00:50:52.605084] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.648426] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:52.655114] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:52.655409] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58337][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:50:52.655944] INFO [LIB] stat (utility.h:1210) [58346][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36EB-0-0] [lt=35] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:52.655984] INFO [LIB] stat (utility.h:1210) [58346][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36EB-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=36706, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:52.656758] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5183855537, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:50:52.656803] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:52.656816] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad908547930) [2025-01-08 00:50:52.656825] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5183855537, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:50:52.657621] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2254, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:52.657675] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=255, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:52.671573] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:52.671666] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=82] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=85) [2025-01-08 00:50:52.672596] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3782) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-01-08 00:50:52.672625] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3805) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=27] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-01-08 00:50:52.675842] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84643-0-0] [lt=255] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2025-01-08 00:50:52.677721] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84643-0-0] [lt=115] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2025-01-08 00:50:52.677762] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1010) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84643-0-0] [lt=38] no transfer partition job [2025-01-08 00:50:52.677777] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1040) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84643-0-0] [lt=10] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer=true) [2025-01-08 00:50:52.680021] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=179] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5723, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:52.680054] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=167, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:52.680134] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=46, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:52.680843] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:284) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84643-0-0] [lt=10] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-01-08 00:50:52.680909] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:493) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84643-0-0] [lt=53] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2025-01-08 00:50:52.680939] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:157) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84643-0-0] [lt=24] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=3, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1003}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone3", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1736324440080711, last_statistic_bg_stat_time=1736326192603087, last_statistic_schema_version=1736326192037600, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-01-08 00:50:52.681828] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485357-0-0] [lt=14] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:50:52.681929] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485357-0-0] [lt=93] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:50:52.698332] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=53] get_max_applied_scn(scn={val:1736326252598201000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43025987}, palf_committed_end_scn_:{val:1736326252598201000, v:0}, last_check_scn_:{val:1736326252598201000, v:0}, max_applied_cb_scn_:{val:1736326252598201000, v:0}}) [2025-01-08 00:50:52.698431] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=93] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43025987}, readable_log_tail:{lsn:43025987}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:24078781, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:52.703844] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE7D7-0-0] [lt=28] submit log success(service_type_=0, last_id=1736326269284252000, limited_id=1736326269284252000) [2025-01-08 00:50:52.703923] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE7D7-0-0] [lt=78] submit log with lock success(service_type_=0, last_id=1736326269284252000, limited_id=1736326269284252000) [2025-01-08 00:50:52.705180] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.705523] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE7D6-0-0] [lt=66] submit log success(service_type_=0, last_id=1736326269254996000, limited_id=1736326269254996000) [2025-01-08 00:50:52.705556] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE7D6-0-0] [lt=32] submit log with lock success(service_type_=0, last_id=1736326269254996000, limited_id=1736326269254996000) [2025-01-08 00:50:52.706405] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=34, stat_interval=1000000, avg (count/cost)=3100, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:52.706433] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=20341) [2025-01-08 00:50:52.706450] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=34, stat_interval=1000000, avg (count/cost)=330, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:52.706479] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=34, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:52.706946] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=51] update id meta(ret=0, service_type=0, limited_id=1736326269284252000, latest_scn={val:1736326252703023001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269284252000, latest_log_ts:{val:1736326252703023001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:50:52.707008] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=62] handle submit callback(service_type_=0, success=true, used_ts=3166, log_ts={val:1736326252703023001, v:0}, limited_id=1736326269284252000) [2025-01-08 00:50:52.707021] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=12] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:50:52.707030] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:50:52.709685] INFO [LIB] stat (utility.h:1210) [58349][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA674E-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=34, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:52.709712] INFO [LIB] stat (utility.h:1210) [58349][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA674E-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=34, stat_interval=1000000, avg (count/cost)=6788, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:52.709724] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58349][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA674E-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=34, avg_log_gen_to_freeze_time=1039, avg_log_gen_to_submit_time=1115, avg_log_submit_to_flush_time=3516, avg_log_submit_to_slide_time=5672) [2025-01-08 00:50:52.709876] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=18] update id meta(ret=0, service_type=0, limited_id=1736326269254996000, latest_scn={val:1736326252704413001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:50:52.710105] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=228] handle submit callback(service_type_=0, success=true, used_ts=4583, log_ts={val:1736326252704413001, v:0}, limited_id=1736326269254996000) [2025-01-08 00:50:52.710119] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=13] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:50:52.710128] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:50:52.710155] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=20341, hit_cnt=34, read_cnt=34, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:52.712215] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1001, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:17618, max_lsn:2144429, max_scn:{val:1736326252410054002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2144429}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2144307}, last_submit_end_lsn:{lsn:2144429}, last_submit_log_id:17618, last_submit_log_pid:1, max_flushed_lsn:{lsn:2144307}, max_flushed_end_lsn:{lsn:2144429}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2144429}, last_slide_log_id:17618, last_slide_scn:{val:1736326252410054002, v:0}, last_slide_lsn:{lsn:2144307}, last_slide_end_lsn:{lsn:2144429}, last_slide_log_pid:1, last_slide_log_accum_checksum:692920881, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924f6a530}, [StateMgr]={this:0x2ad924f6e830, self:"192.168.159.137:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:17619, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324447115984, palf_role_change_cb:0x2ad924f752f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924f6bc70}, [ModeMgr]={palf_id:1001, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324447762546, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436756804001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/meta", dir_fd:177, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924f70a38, flashback_version:0}, log_storage:{log_tail:{lsn:2144429}, readable_log_tail:{lsn:2144429}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64960339, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1736324447114985, this:0x2ad924f6ef70}, [Reconfirm]={palf_id:1001, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:1"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324447117136, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924f6e9b0}) [2025-01-08 00:50:52.712727] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=505] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326252657702, last_flushed_end_lsn:{lsn:2144429}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326252659759, last_flushed_end_lsn:{lsn:2144429}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326252660886, last_flushed_end_lsn:{lsn:2144429}}]) [2025-01-08 00:50:52.712820] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=88] current log_task status(palf_id=1001, self="192.168.159.137:2882", log_id=17619, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326252657702, last_flushed_end_lsn:{lsn:2144429}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326252659759, last_flushed_end_lsn:{lsn:2144429}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326252660886, last_flushed_end_lsn:{lsn:2144429}}]) [2025-01-08 00:50:52.741420] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:52.761059] INFO [STORAGE] try_shrink (ob_compaction_memory_pool.cpp:522) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=81] Compaction Mem Pool current stat: (ret=0, max_block_num=16, total_block_num=8, Chunk list size: =1) [2025-01-08 00:50:52.766654] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:52.768150] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=30] ====== tenant freeze timer task ====== [2025-01-08 00:50:52.774316] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:50:52.776657] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4429-0-0] [lt=42] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:52.776687] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4429-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:52.777726] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:52.780292] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:52.793312] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D008-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:52.803571] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3813-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10190) [2025-01-08 00:50:52.803940] WDIAG [PALF] leader_get_committed_log_info_ (log_sliding_window.cpp:4184) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3813-0-0] [lt=365][errcode=-4233] get_log_task failed(ret=-4233, palf_id=1, self="192.168.159.137:2882", max_log_id=32304, log_task=NULL) [2025-01-08 00:50:52.804077] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=53] BlockGCTimerTask success(ret=0, cost_time_us=16, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:50:52.804836] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=39] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:50:52.805447] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=71][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.806637] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=349] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:52.849010] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=51] PNIO [ratelimit] time: 1736326252849006, bytes: 866887259, bw: 0.254562 MB/s, add_ts: 1007586, add_bytes: 268953 [2025-01-08 00:50:52.851250] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=42] ====== traversal_flush timer task ====== [2025-01-08 00:50:52.851516] INFO [MDS] flush (mds_table_mgr.cpp:195) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F85798-0-0] [lt=23] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326252766877000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.851659] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F85798-0-0] [lt=136] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:50:52.851741] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=44] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:50:52.857050] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1003}, {id:1}, {id:1002}]) [2025-01-08 00:50:52.858486] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:50:52.867819] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1736326252766877000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1736326252867782}, version_delta=100941) [2025-01-08 00:50:52.878768] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=193, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:52.878852] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=82] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=4, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:52.895303] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=31] ====== check clog disk timer task ====== [2025-01-08 00:50:52.895333] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:50:52.897421] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.897460] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.900353] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.900441] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:52.901437] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C84-0-0] [lt=56][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:52.902715] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C84-0-0] [lt=42][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:52.905573] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.911785] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:52.926456] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=22] ====== check clog disk timer task ====== [2025-01-08 00:50:52.926484] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:52.936376] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=87] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=34, stat_interval=1000000, avg (count/cost)=69, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:52.941812] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:52.955388] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=64] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:52.956840] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:52.957072] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=9134, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:52.957098] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:52.957121] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=170, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:52.957276] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=79] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=56, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:52.957832] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=74] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:52.958363] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:50:52.958650] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:52.958743] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:50:52.972842] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:52.974521] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:52.977914] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17805, miss_cnt:0, cache_read_size:2177952, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:53.000645] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D2F-0-0] [lt=38] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:53.004030] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=81] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=57, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:53.005939] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.006285] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=34] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:50:53.006308] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=21] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=1) [2025-01-08 00:50:53.006318] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=9] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:50:53.006328] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=6] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:50:53.011401] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:53.011453] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:35, bw:162635, [delay/us]:prepare:7, schedule:96, submit:1518, rt:1344, total:2967) [2025-01-08 00:50:53.011468] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:53.011521] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:35, avg_bw:162635, [delay/us]: prepare:7, schedule:96, submit:1518, device:1344, total:2967) [2025-01-08 00:50:53.011534] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:53.011572] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:12288 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:17, bw:77252, [delay/us]:prepare:15, schedule:137, submit:126, rt:1151, total:1430) [2025-01-08 00:50:53.011582] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:53.011618] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:77258, [delay/us]: prepare:15, schedule:137, submit:126, device:1151, total:1430) [2025-01-08 00:50:53.011628] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:53.011650] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:7, bw:32529, [delay/us]:prepare:14, schedule:74, submit:1, rt:1403, total:1493) [2025-01-08 00:50:53.011673] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4198, iops:40, bw:166715, [delay/us]:prepare:7, schedule:2512, submit:492, rt:1591, total:4603) [2025-01-08 00:50:53.011682] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12198, [delay/us]:prepare:1, schedule:567, submit:263, rt:238, total:1071) [2025-01-08 00:50:53.011690] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:53.011717] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4681, avg_iops:7, avg_bw:32529, [delay/us]: prepare:14, schedule:74, submit:1, device:1403, total:1493) [2025-01-08 00:50:53.011733] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12198, [delay/us]: prepare:1, schedule:567, submit:263, device:238, total:1071) [2025-01-08 00:50:53.011740] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4198, avg_iops:40, avg_bw:166712, [delay/us]: prepare:7, schedule:2512, submit:492, device:1591, total:4603) [2025-01-08 00:50:53.011750] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:53.011769] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4066, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:462, total:469) [2025-01-08 00:50:53.011793] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:53.011820] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4066, [delay/us]: prepare:0, schedule:0, submit:600000000, device:462, total:469) [2025-01-08 00:50:53.011837] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:53.011852] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:53.011861] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:53.011869] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:53.011877] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=272kB/s, net_out=294kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:53.014009] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB50-0-0] [lt=56][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193757927842, cache_obj->added_lc()=false, cache_obj->get_object_id()=13556, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:53.014966] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43028125}], scn_array=[cnt:1, {val:1736326252995008000, v:0}], curr_size=589, accum_size=5243, time_cost=10472) [2025-01-08 00:50:53.031279] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B8B-0-0] [lt=23] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:53.031419] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B8D-0-0] [lt=34] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:53.034305] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=33] runTimer to refresh locality_info(ret=0) [2025-01-08 00:50:53.036884] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:346) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=37] set_server_locality_array success(server_locality_array=[{inited:true, addr:"192.168.159.137:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322580423131, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.138:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581270010, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.139:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581175973, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-01-08 00:50:53.036937] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=39] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-01-08 00:50:53.036949] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=11] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-01-08 00:50:53.036965] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1001, primary_zone=RANDOM) [2025-01-08 00:50:53.036972] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=7] add_locality_zone success(locality_zone={tenant_id:1001, region_priority:18446744073709551615}) [2025-01-08 00:50:53.036984] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1002, primary_zone=RANDOM) [2025-01-08 00:50:53.036991] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=7] add_locality_zone success(locality_zone={tenant_id:1002, region_priority:18446744073709551615}) [2025-01-08 00:50:53.036999] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1001, region_priority:18446744073709551615}, {tenant_id:1002, region_priority:18446744073709551615}]}) [2025-01-08 00:50:53.038707] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58244][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3F-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326253140213, keepalive_ts:-1}) [2025-01-08 00:50:53.038732] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58244][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3F-0-0] [lt=24] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326253140213, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:53.042296] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185110-0-0] [lt=11] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326248135121, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:50:53.042358] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185110-0-0] [lt=58] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326248137150, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:50:53.042384] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185110-0-0] [lt=24] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, renew_time:1736326248139373, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, renew_time:1736326248139375, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, renew_time:1736326248139376, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326248139369, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:50:53.055972] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36F3-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:53.056988] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:50:53.057017] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:50:53.058427] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7623, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:53.058546] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=115] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=309, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:53.058558] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:53.060438] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:53.061424] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:53.067991] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D1728528A-0-0] [lt=39] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:50:53.073273] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:53.083028] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:53.106083] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.112298] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C291-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:53.112364] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C291-0-0] [lt=49] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326252986041000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326252986041000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:53.112961] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=83] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326249105334, last_first_register_time_us=1736324454612308, register_time_us=1736326249105334, curr_time_us=1736326253112958) [2025-01-08 00:50:53.113028] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326253112995, last_submit_register_req_time_us=1736326253112995, register_parent_reason=1) [2025-01-08 00:50:53.113044] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:50:53.125023] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=61] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:53.138581] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=50] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8564, offset=2429274}, end_cursor:ObLogCursor{file_id=1, log_id=8565, offset=2429579}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946bc050, len:3750, log_data_len:305, seq:8564, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:53.139405] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3E-0-0] [lt=93] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:53.139466] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3E-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326252911332001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326252911332001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:53.140329] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=19476) [2025-01-08 00:50:53.142409] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:53.144088] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=34] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=408941, clean_num=31457) [2025-01-08 00:50:53.148654] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:53.151130] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E3E-0-0] [lt=50] get number(ret=0, service_type_=0, range=1, base_id=1736326253150512000, start_id=1736326253150512000, end_id=1736326253150512001) [2025-01-08 00:50:53.156853] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=47] ====== check clog disk timer task ====== [2025-01-08 00:50:53.156916] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=50] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:53.158714] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=59] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8565, offset=2429579}, end_cursor:ObLogCursor{file_id=1, log_id=8566, offset=2429876}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946c4050, len:3445, log_data_len:297, seq:8565, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:53.161811] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3E-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:53.161862] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3E-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326252911332001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326252911332001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:53.171704] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FA1-0-0] [lt=53] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=17, avg_log_submit_to_flush_time=10467, avg_log_submit_to_slide_time=31510) [2025-01-08 00:50:53.171787] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=58] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:53.171820] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17720, miss_cnt:0, cache_read_size:2167749, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:53.173807] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC371-0-0] [lt=50][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:53.175045] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58339][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3819-0-0] [lt=24] init point thread id with(&point=0x5572d50e2200, idx_=995, point=[thread id=58339, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58339) [2025-01-08 00:50:53.176471] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=26] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:420K/97:1083619 disk:420K/97:1083619 t1m2:160K/35:101320 t1m2g0:160K/35:101320 t1002m2:180K/44:2282447 t1002m2g0:180K/44:2282447 t1001m2:80K/18:63174 t1001m2g0:80K/18:63174 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:53.177423] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=55] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1085232, replace_num=15728) [2025-01-08 00:50:53.177524] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=99] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:53.180602] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=38] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8566, offset=2429876}, end_cursor:ObLogCursor{file_id=1, log_id=8567, offset=2430173}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946c8050, len:3148, log_data_len:297, seq:8566, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:53.182079] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=91] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8567, offset=2430173}, end_cursor:ObLogCursor{file_id=1, log_id=8568, offset=2430470}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946d0050, len:2851, log_data_len:297, seq:8567, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:53.184947] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1736326253183295000, start_id=1736326253183295000, end_id=1736326253183295001) [2025-01-08 00:50:53.196421] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=9] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:53.197879] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=53] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:53.198087] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=47] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326244258401000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:53.198192] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=101] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326244258401000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326244258401000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:53.205530] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:53.206424] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=146][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.215271] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:50:53.222895] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=207][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:53.222938] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:53.226170] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8701F-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:53.226561] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=234][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:53.226610] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:53.248956] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=19] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:53.248979] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=22] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:53.249033] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E2-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:53.249052] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E2-0-0] [lt=15] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:53.253079] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:202) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=7] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2025-01-08 00:50:53.253124] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=39] finish to resize log external storage handler(current_ts=1736326253253119, tenant_max_cpu=4, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:50:53.253139] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=13] snapshot_traffic_info once [2025-01-08 00:50:53.253901] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:53.255166] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=45] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:53.255346] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=51] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245675194000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:53.277689] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=62] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-01-08 00:50:53.277809] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=38] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:53.279432] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC835C-0-0] [lt=28] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=6800, avg_log_submit_to_slide_time=28900) [2025-01-08 00:50:53.279531] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:53.281671] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=62] gc stale ls task succ [2025-01-08 00:50:53.281783] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:53.281870] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=80] temporary file module use no blocks [2025-01-08 00:50:53.281882] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:53.281896] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:53.281963] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=64] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193757195475, flush_io_finished_ret_:0, flush_io_finished_round_:1916, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:53.286384] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9B-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:53.286416] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9B-0-0] [lt=29] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:53.287609] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46498][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB52-0-0] [lt=5][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193758201417, cache_obj->added_lc()=false, cache_obj->get_object_id()=13557, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:53.297526] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185410-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:53.306503] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.308781] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBD-0-0] [lt=18] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326253411670, keepalive_ts:-1}) [2025-01-08 00:50:53.308881] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBD-0-0] [lt=98] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326253411670, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:53.319240] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.319298] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.335521] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=52] current monitor number(seq_=-1) [2025-01-08 00:50:53.340426] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=26, avg_log_batch_cnt=1, total_group_log_size=13075, avg_group_log_size=502) [2025-01-08 00:50:53.340454] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=13065) [2025-01-08 00:50:53.340471] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=28, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:53.350698] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=62] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:53.351080] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:53.352523] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:53.352614] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245564663000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:53.357574] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=80, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:53.359816] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:50:53.370754] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690D57-0-0] [lt=33] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.796639385223388672e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=898, Throttle Trigger(MB)=538, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.733554252408669072e+01) [2025-01-08 00:50:53.375785] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:50:53.378806] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:50:53.379388] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58349][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:50:53.387504] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=1981, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:53.387555] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=50] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9762) [2025-01-08 00:50:53.387580] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=513, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:53.387644] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=13, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:53.389286] INFO [LIB] stat (utility.h:1210) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA381D-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=21, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:53.389327] INFO [LIB] stat (utility.h:1210) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA381D-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4001, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:53.389341] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA381D-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=82, avg_log_submit_to_flush_time=2567, avg_log_submit_to_slide_time=3919) [2025-01-08 00:50:53.389484] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9762, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:53.390112] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326252704413003, v:0}, new_max_version={val:1736326252704413003, v:0}, last_min_version={val:1736326252486474004, v:0}, last_max_version={val:1736326252486474004, v:0}, total_time_us=5603, delta_us=685697) [2025-01-08 00:50:53.390161] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326252704413003, v:0}, min_version={val:1736326252704413003, v:0}, max_version={val:1736326252704413003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:53.391490] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=50] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:53.398374] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:53.398454] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=73] temporary file module use no blocks [2025-01-08 00:50:53.398467] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:53.398481] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:53.398495] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193757310949, flush_io_finished_ret_:0, flush_io_finished_round_:3740, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:53.406643] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.428913] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=81][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:50:53.429058] WDIAG read_multi_row (ob_table_access_helper.h:238) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:50:53.430885] INFO delete_row (ob_table_access_helper.h:339) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=39] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1001, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326253429097) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326253429097) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:50:53.453669] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BEC2-0-0] [lt=81] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:50:53.458410] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.458497] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=50] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:53.458534] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:53.459726] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.467936] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E7-0-0] [lt=80] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:53.467995] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E7-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326253203138001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326253203138001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:53.468714] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:194) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=67] get wrs ts(ls_id={id:1}, delta=97710, log_service_ts=1736326253370559, min_tx_service_ts=4611686018427387, timestamp={val:1736326253370559000, v:0}) [2025-01-08 00:50:53.468749] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=58, Other_Error_Cnt=0, Submit_Succ_Cnt=42, last_scn="{val:1736326248747804001, v:0}", last_lsn={lsn:84153684}, last_gts={val:1736326253411478000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-01-08 00:50:53.471406] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] ====== tenant freeze timer task ====== [2025-01-08 00:50:53.471447] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=36] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70075580, tenant_id=1001) [2025-01-08 00:50:53.471471] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:53.477704] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4437-0-0] [lt=86] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=26, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:53.477731] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4437-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=34701, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:53.478704] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=69] clear dtl interm result cost(us)(clear_cost=8094, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:50:53.484587] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=77] ====== tenant freeze timer task ====== [2025-01-08 00:50:53.484766] INFO [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1165) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] refresh queuing mode(ret=0, tenant_id=1, stream_cnt=16, update_schema_cnt=16, cost_time=58) [2025-01-08 00:50:53.484792] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1215) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=23] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1, interval_step=1, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1, bucket_cnt:0, tenant_mem_user_limit:1073741824, tenant_mem_user_hold:213045248, cached_node_array:0x2ad89c1e7b70}) [2025-01-08 00:50:53.488485] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F474-0-0] [lt=103] submit log success(service_type_=0, last_id=1736326270087570000, limited_id=1736326270087570000) [2025-01-08 00:50:53.488509] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F474-0-0] [lt=23] submit log with lock success(service_type_=0, last_id=1736326270087570000, limited_id=1736326270087570000) [2025-01-08 00:50:53.491338] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] update id meta(ret=0, service_type=0, limited_id=1736326270087570000, latest_scn={val:1736326253487654001, v:0}, *this={tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326250908389001, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326270087570000, latest_log_ts:{val:1736326253487654001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:50:53.491423] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=83] handle submit callback(service_type_=0, success=true, used_ts=2939, log_ts={val:1736326253487654001, v:0}, limited_id=1736326270087570000) [2025-01-08 00:50:53.491439] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=15] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:50:53.491448] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:50:53.492925] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:53.503992] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=21] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:50:53.506823] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.514056] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1736326253514052, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007684, add_bytes: 0 [2025-01-08 00:50:53.518808] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.542448] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.543158] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:53.544332] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.544634] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58327][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=92] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:50:53.554955] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:53.558028] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=59, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:53.558079] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:53.558102] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:53.558117] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:53.559075] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:53.560773] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:53.562118] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FAD-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:53.562521] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FAD-0-0] [lt=401] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4309, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:53.564245] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8365-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:53.564327] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8365-0-0] [lt=80] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4658, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:53.564339] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8365-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=103, avg_log_submit_to_flush_time=1918, avg_log_submit_to_slide_time=4554) [2025-01-08 00:50:53.567840] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=148] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1348, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:53.573629] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:53.577919] INFO [LIB] stat (utility.h:1210) [58346][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4439-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:53.577990] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58346][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4439-0-0] [lt=47][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17898, log_task={header:{begin_lsn:{lsn:2178440}, end_lsn:{lsn:2178562}, log_id:17898, min_scn:{val:1736326252986041004, v:0}, max_scn:{val:1736326252986041004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2178318}, prev_proposal_id:1, committed_end_lsn:{lsn:2178440}, data_checksum:1715074492, accum_checksum:4019080202, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326253477613, freeze_ts:1736326253477613, submit_ts:1736326253477625, flushed_ts:1736326253481044, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:3419}, fs_cb_begin_ts=1736326253577981, log_life_time=100368) [2025-01-08 00:50:53.579017] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:50:53.583225] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=84] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:53.587337] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=67] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=9763, avg_group_log_size=542) [2025-01-08 00:50:53.587366] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9762) [2025-01-08 00:50:53.604451] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] no need to update min/max version(new_version={val:1736326253203138000, v:0}, min_version={val:1736326253203138000, v:0}, max_version={val:1736326253203138000, v:0}) [2025-01-08 00:50:53.606923] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.632981] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B8F-0-0] [lt=11] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:53.648946] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:53.657793] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5185675255, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:50:53.657907] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=38] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:53.657923] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5185675255, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:50:53.659646] INFO [LIB] stat (utility.h:1210) [58244][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36FF-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:53.659875] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1757, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:53.659917] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58244][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36FF-0-0] [lt=259] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1333, avg_group_log_size=121) [2025-01-08 00:50:53.659929] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=105, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:53.660243] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:53.660312] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58329][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=40] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:50:53.663044] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D64A-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=34, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:53.663288] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D64A-0-0] [lt=143] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4379, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:53.663510] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB APPEND COST TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=160, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:50:53.663546] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=35] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=4719, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:50:53.663557] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:50:53.663564] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=-554, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:50:53.663571] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=5] [APPLY STAT CB TOTAL TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=4325, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:50:53.677030] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=33] BlockGCTimerTask success(ret=0, cost_time_us=30, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):279, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:50:53.704592] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326253488901000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=215675, in_cluster_service=true, cluster_version={val:1736326253203138000, v:0}, min_cluster_version={val:1736326253203138000, v:0}, max_cluster_version={val:1736326253203138000, v:0}, get_cluster_version_err=0, cluster_version_delta=501438, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18544, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=599341359, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:53.706543] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46508][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDBB-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193758620283, cache_obj->added_lc()=false, cache_obj->get_object_id()=13558, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:53.707266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.707842] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):279, sequence:1}) [2025-01-08 00:50:53.708603] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=32] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:50:53.710235] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326253709530000, start_id=1736326253709530000, end_id=1736326253709530001) [2025-01-08 00:50:53.715757] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=17086819, this={dio_aligned_buf:{buf_write_offset:2948, buf_padding_size:1148, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:70311, truncate_used_ts:17326}, log_block_size:67108864, io_fd:{first_id:-1, second_id:152, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=17084416, buf_len=545, total_write_size=55701, total_write_size_after_dio=573440, ob_pwrite_used_ts=321364, count=127, each_pwrite_cost=2530) [2025-01-08 00:50:53.715799] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1775, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:53.715812] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4085) [2025-01-08 00:50:53.715830] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=182, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:53.715881] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:53.715900] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3597, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:53.715909] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=93, avg_log_submit_to_flush_time=1988, avg_log_submit_to_slide_time=3503) [2025-01-08 00:50:53.715929] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:53.716003] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4085, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:53.717964] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:50:53.717967] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58351][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:53.745740] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:50:53.745814] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=73] try gc retain ctx [2025-01-08 00:50:53.745825] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-01-08 00:50:53.745860] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326253696901000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326243676959001, v:0}, keep_alive_scn_:{val:1736326248586728001, v:0}, update_ts_:1736326253745393}) [2025-01-08 00:50:53.746589] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269284252000, latest_log_ts:{val:1736326252703023001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:53.751131] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:263) [50686][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1FF8458E-0-0] [lt=24] ObArchiveSender is running(thread_index=3) [2025-01-08 00:50:53.754771] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=39] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:50:53.754947] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EC8-0-0] [lt=11] InfoPoolResizeTask(cost_ts=8) [2025-01-08 00:50:53.755545] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=51] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:53.766824] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:53.778024] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:53.778674] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A443D-0-0] [lt=17] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:53.778739] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A443D-0-0] [lt=61] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:53.780036] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:50:53.784270] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1985, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:53.784304] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:53.784325] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=397, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:53.784416] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:53.785324] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326253203138004, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=582175, in_cluster_service=true, cluster_version={val:1736326252911332004, v:0}, min_cluster_version={val:1736326252911332004, v:0}, max_cluster_version={val:1736326252911332004, v:0}, get_cluster_version_err=0, cluster_version_delta=873981, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18108, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=481702840, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:53.796601] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D00B-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:53.807504] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.811055] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=44] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:53.814722] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] LogLoopThread round_cost_time(us)(round_cost_time=24) [2025-01-08 00:50:53.814776] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:19261, max_lsn:18644718, max_scn:{val:1736326253487654004, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:18644718}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:18644596}, last_submit_end_lsn:{lsn:18644718}, last_submit_log_id:19261, last_submit_log_pid:1, max_flushed_lsn:{lsn:18644596}, max_flushed_end_lsn:{lsn:18644718}, max_flushed_log_pid:1, committed_end_lsn:{lsn:18644718}, last_slide_log_id:19261, last_slide_scn:{val:1736326253487654004, v:0}, last_slide_lsn:{lsn:18644596}, last_slide_end_lsn:{lsn:18644718}, last_slide_log_pid:1, last_slide_log_accum_checksum:3817258203, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8fc454530}, [StateMgr]={this:0x2ad8fc458830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:19262, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324399554770, palf_role_change_cb:0x2ad8fc45f2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8fc455c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324399758968, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/meta", dir_fd:175, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8fc45aa38, flashback_version:0}, log_storage:{log_tail:{lsn:18644718}, readable_log_tail:{lsn:18644718}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48460050, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324399554279, this:0x2ad8fc458f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324399555093, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8fc4589b0}) [2025-01-08 00:50:53.815105] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=322] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326253760153, last_flushed_end_lsn:{lsn:18644718}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326253762064, last_flushed_end_lsn:{lsn:18644718}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326253763501, last_flushed_end_lsn:{lsn:18644718}}]) [2025-01-08 00:50:53.815130] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=19262, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326253760153, last_flushed_end_lsn:{lsn:18644718}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326253762064, last_flushed_end_lsn:{lsn:18644718}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326253763501, last_flushed_end_lsn:{lsn:18644718}}]) [2025-01-08 00:50:53.815375] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1002, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:17901, max_lsn:2178928, max_scn:{val:1736326253487654002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2178928}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2178806}, last_submit_end_lsn:{lsn:2178928}, last_submit_log_id:17901, last_submit_log_pid:1, max_flushed_lsn:{lsn:2178806}, max_flushed_end_lsn:{lsn:2178928}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2178928}, last_slide_log_id:17901, last_slide_scn:{val:1736326253487654002, v:0}, last_slide_lsn:{lsn:2178806}, last_slide_end_lsn:{lsn:2178928}, last_slide_log_pid:1, last_slide_log_accum_checksum:1331969433, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad92435e530}, [StateMgr]={this:0x2ad924362830, self:"192.168.159.137:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.138:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:17902, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad9243692f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326252110999, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326252110999, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad92435fc70}, [ModeMgr]={palf_id:1002, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436803442001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/meta", dir_fd:184, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924364a38, flashback_version:0}, log_storage:{log_tail:{lsn:2178928}, readable_log_tail:{lsn:2178928}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64925840, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:-1, this:0x2ad924362f70}, [Reconfirm]={palf_id:1002, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9243629b0}) [2025-01-08 00:50:53.815597] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=217][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:50:53.815611] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:50:53.815621] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:50:53.815630] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] current log_task status(palf_id=1002, self="192.168.159.137:2882", log_id=17902, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:50:53.844525] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:50:53.844574] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=46] try gc retain ctx [2025-01-08 00:50:53.844584] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-01-08 00:50:53.844609] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1736326253770280000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84188622}, palf_committed_end_scn_:{val:1736326253770280000, v:0}, last_check_scn_:{val:1736326253770280000, v:0}, max_applied_cb_scn_:{val:1736326253770280000, v:0}}) [2025-01-08 00:50:53.844636] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84188622}, readable_log_tail:{lsn:84188622}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:50020914, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:53.844735] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=95] finish update min start scn(max_decided_scn={val:1736326253770280000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326243686095001, v:0}, keep_alive_scn_:{val:1736326248747804001, v:0}, update_ts_:1736326253843814}) [2025-01-08 00:50:53.845464] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:53.848477] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=17] ls blacklist refresh finish(cost_time=15019) [2025-01-08 00:50:53.856638] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1736326253856634, bytes: 867205592, bw: 0.301288 MB/s, add_ts: 1007628, add_bytes: 318333 [2025-01-08 00:50:53.857403] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1001}, {id:1002}]) [2025-01-08 00:50:53.859676] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1348) [2025-01-08 00:50:53.875815] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=140] LogLoopThread round_cost_time(us)(round_cost_time=16) [2025-01-08 00:50:53.881577] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=13] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=207, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:53.881668] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=90] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:53.897059] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=66] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.897529] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.897581] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.897608] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=61] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:53.907751] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:53.908899] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9038) [2025-01-08 00:50:53.911088] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58353][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:53.911232] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58353][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3824-0-0] [lt=26] init point thread id with(&point=0x5572d50e2580, idx_=1009, point=[thread id=58353, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58353) [2025-01-08 00:50:53.912298] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:50:53.913622] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326253913589], gts:1736326253912776000, latest_srr:[mts=1736326253913589]}) [2025-01-08 00:50:53.913701] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=77] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:53.913728] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326253913714], gts:1736326253912776000, latest_srr:[mts=1736326253913714]}) [2025-01-08 00:50:53.914060] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=513, get_gts_with_stc_cnt=3, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=3, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:53.914101] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=40] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326253914027], gts:1736326253912776000, latest_srr:[mts=1736326253914027]}) [2025-01-08 00:50:53.915473] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C85-0-0] [lt=30][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:53.917299] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C85-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:53.932340] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:50:53.944716] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=122] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=75, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:53.956061] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:53.958467] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:53.959001] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:53.959034] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:53.960114] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1348) [2025-01-08 00:50:53.960411] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:53.960554] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=177] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:53.961434] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1429, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:53.961476] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:50:53.961496] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=256, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:53.961722] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=64, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:53.980390] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:50:53.980554] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17815, miss_cnt:0, cache_read_size:2179172, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:54.005477] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=90] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:54.006755] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:50:54.007897] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:54.008721] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=47] get number(ret=0, service_type_=0, range=1, base_id=1736326254008054000, start_id=1736326254008054000, end_id=1736326254008054001) [2025-01-08 00:50:54.008894] INFO [LIB] stat (utility.h:1210) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=48] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=88, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:54.010388] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=47] ====== checkpoint timer task ====== [2025-01-08 00:50:54.011062] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=38] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:1, active_tx_count:1, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:54.011125] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=61] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1736326254008054000, v:0}) [2025-01-08 00:50:54.011419] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=27] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269284252000, latest_log_ts:{val:1736326252703023001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:54.011783] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=360] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:50:54.011903] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=112] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:54.011939] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=33] get rec log scn(service_type_=0, rec_log_ts={val:1736326252703023001, v:0}) [2025-01-08 00:50:54.011949] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:50:54.011957] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:50:54.011968] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:54.011992] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=19] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:54.012022] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:54.012045] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:54.012066] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:54.012075] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:50:54.018033] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:54.018107] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48827, [delay/us]:prepare:8, schedule:98, submit:156, rt:1598, total:1860) [2025-01-08 00:50:54.018125] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:54.018259] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=62] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48823, [delay/us]: prepare:8, schedule:98, submit:156, device:1598, total:1860) [2025-01-08 00:50:54.018275] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:54.018342] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:81369, [delay/us]:prepare:14, schedule:45, submit:97, rt:1806, total:1964) [2025-01-08 00:50:54.018355] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:54.018398] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:81368, [delay/us]: prepare:14, schedule:45, submit:97, device:1806, total:1964) [2025-01-08 00:50:54.018429] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:54.018455] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16273, [delay/us]:prepare:4, schedule:312, submit:3, rt:1004, total:1324) [2025-01-08 00:50:54.018478] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4195, iops:41, bw:170869, [delay/us]:prepare:13, schedule:107, submit:162, rt:1294, total:1579) [2025-01-08 00:50:54.018487] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:54.018516] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16273, [delay/us]: prepare:4, schedule:312, submit:3, device:1004, total:1324) [2025-01-08 00:50:54.018531] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4196, avg_iops:41, avg_bw:170871, [delay/us]: prepare:13, schedule:107, submit:162, device:1294, total:1579) [2025-01-08 00:50:54.018540] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:54.018558] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4068, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:264, total:272) [2025-01-08 00:50:54.018582] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:54.018617] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4068, [delay/us]: prepare:0, schedule:0, submit:600000000, device:264, total:272) [2025-01-08 00:50:54.018734] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=103] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:54.018756] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:54.018766] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:54.018775] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:54.018783] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=293kB/s, net_out=255kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:54.058610] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1348, avg_group_log_size=122) [2025-01-08 00:50:54.058654] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1348) [2025-01-08 00:50:54.059395] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3707-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:50:54.059528] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3707-0-0] [lt=116][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17817, log_task={header:{begin_lsn:{lsn:2168594}, end_lsn:{lsn:2168716}, log_id:17817, min_scn:{val:1736326253707292002, v:0}, max_scn:{val:1736326253707292002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2168472}, prev_proposal_id:1, committed_end_lsn:{lsn:2168594}, data_checksum:940009244, accum_checksum:3725933102, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326253959380, freeze_ts:1736326253959380, submit_ts:1736326253959387, flushed_ts:1736326253961611, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:2224}, fs_cb_begin_ts=1736326254059517, log_life_time=100137) [2025-01-08 00:50:54.060406] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1644, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:54.060535] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=119] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=86, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:54.060634] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:54.061163] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:54.063103] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=58] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:54.073844] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=19] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:54.083424] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:54.108486] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1736326254107525000, start_id=1736326254107525000, end_id=1736326254107525001) [2025-01-08 00:50:54.109072] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:54.134964] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C292-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:54.135042] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C292-0-0] [lt=60] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326253988817000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326253988817000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:54.147897] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4076) [2025-01-08 00:50:54.149098] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=53] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:54.156974] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:54.159457] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3709-0-0] [lt=35][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17818, log_task={header:{begin_lsn:{lsn:2168716}, end_lsn:{lsn:2168838}, log_id:17818, min_scn:{val:1736326253707292003, v:0}, max_scn:{val:1736326253707292003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2168594}, prev_proposal_id:1, committed_end_lsn:{lsn:2168716}, data_checksum:940009244, accum_checksum:3079701573, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326254059357, freeze_ts:1736326254059357, submit_ts:1736326254059363, flushed_ts:1736326254061501, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:2138}, fs_cb_begin_ts=1736326254159446, log_life_time=100089) [2025-01-08 00:50:54.171694] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC374-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:54.172843] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=183] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:50:54.174829] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326254174436000, start_id=1736326254174436000, end_id=1736326254174436001) [2025-01-08 00:50:54.179269] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:300K/70:107420 disk:300K/70:107420 t1m2:48K/11:69355 t1m2g0:48K/11:69355 t1002m2:184K/44:121176 t1002m2g0:184K/44:121176 t1001m2:68K/15:94984 t1001m2g0:68K/15:94984 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:54.180838] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=62] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:50:54.192248] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8568, offset=2430470}, end_cursor:ObLogCursor{file_id=1, log_id=8569, offset=2430775}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946cc050, len:2554, log_data_len:305, seq:8568, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:54.192920] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3F-0-0] [lt=127] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:54.193008] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3F-0-0] [lt=72] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326254174436000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326254174436000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:54.201115] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=62] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:50:54.203638] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8569, offset=2430775}, end_cursor:ObLogCursor{file_id=1, log_id=8570, offset=2431072}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946d4050, len:2249, log_data_len:297, seq:8569, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:54.204189] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3F-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:54.204373] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3F-0-0] [lt=111] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326253912776002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326253912776002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:54.206112] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE7E1-0-0] [lt=8] get number(ret=0, service_type_=0, range=1, base_id=1736326254205134000, start_id=1736326254205134000, end_id=1736326254205134001) [2025-01-08 00:50:54.209013] INFO [LIB] stat (utility.h:1210) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E3F-0-0] [lt=47] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:54.209327] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:54.209761] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE7E0-0-0] [lt=9] get number(ret=0, service_type_=0, range=1, base_id=1736326254208689000, start_id=1736326254208689001, end_id=1736326254208689002) [2025-01-08 00:50:54.214030] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=40] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8570, offset=2431072}, end_cursor:ObLogCursor{file_id=1, log_id=8571, offset=2431369}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946d8050, len:1952, log_data_len:297, seq:8570, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:54.215179] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8571, offset=2431369}, end_cursor:ObLogCursor{file_id=1, log_id=8572, offset=2431666}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946dc050, len:1655, log_data_len:297, seq:8571, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:54.215462] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=40] gc stale ls task succ [2025-01-08 00:50:54.215461] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17730, miss_cnt:0, cache_read_size:2168960, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:54.215516] INFO [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:197) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] skip repeat study(ret=0, this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326194148156, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}, current_timestamp=1736326254215514, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:50:54.217151] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=25] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325654677140000, snapshot_type=1, create_time=1736325654678024, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:50:54.217177] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325662041557001, snapshot_type=1, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:50:54.217188] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325663340298000, snapshot_type=1, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:50:54.217271] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=77] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325654363217000, snapshot_type=2, create_time=1736325654678024, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:50:54.217281] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325661073141000, snapshot_type=2, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:50:54.217291] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325662946686001, snapshot_type=2, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:50:54.217300] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325654634574000, snapshot_type=3, create_time=1736325654678024, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:50:54.217309] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325661436394000, snapshot_type=3, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:50:54.217330] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325663119496000, snapshot_type=3, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:50:54.217339] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325654678024, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:50:54.217349] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:50:54.217358] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:50:54.217681] INFO [MVCC] decide_reserved_snapshot_version_ (ob_multi_version_garbage_collector.cpp:620) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] succeed to update global reserved snapshot(ret=0, this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326194148156, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}, global_reserved_snapshot_={val:1736325654363217000, v:0}, reserved_snapshot={val:1736325654363217000, v:0}) [2025-01-08 00:50:54.217703] INFO [MVCC] refresh_ (ob_multi_version_garbage_collector.cpp:554) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=21] multi-version garbage collector refresh successfully(this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326194148156, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}, collector={reserved_snapshot_version:{val:1736325654363217000, v:0}, reserved_snapshot_type:2, reserved_status:-1, reserved_create_time:1736325654678024, reserved_addr:"192.168.159.137:2882", is_this_server_disabled:false, status:0}) [2025-01-08 00:50:54.217721] INFO [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:232) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] repeat refresh successfully(ret=0, this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}, current_timestamp=1736326254215541, GARBAGE_COLLECT_RETRY_INTERVAL=60000000) [2025-01-08 00:50:54.217733] INFO [MVCC] repeat_reclaim (ob_multi_version_garbage_collector.cpp:267) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] skip repeat reclaim(ret=0, this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}, current_timestamp=1736326254217733, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:50:54.222867] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:54.222925] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:54.226368] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87025-0-0] [lt=8][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:54.226444] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:54.226456] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:54.262645] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58333][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D656-0-0] [lt=61] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=31, avg_log_submit_to_flush_time=1720, avg_log_submit_to_slide_time=21286) [2025-01-08 00:50:54.262735] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:54.278076] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=28] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:54.281745] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=75] gc stale ls task succ [2025-01-08 00:50:54.288021] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9C-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:54.288053] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9C-0-0] [lt=29] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:54.300634] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185411-0-0] [lt=95][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:54.309153] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46500][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB54-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193759222983, cache_obj->added_lc()=false, cache_obj->get_object_id()=13559, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:54.309865] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=73] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:54.309907] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:54.316149] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=47] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:50:54.316190] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=35] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:50:54.316203] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:50:54.316214] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:54.318780] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:54.318819] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:54.318870] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=78] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:50:54.319987] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=1112] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:54.334653] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD85-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326254440618, keepalive_ts:-1}) [2025-01-08 00:50:54.334691] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD85-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326254440618, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:54.335791] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=44] current monitor number(seq_=-1) [2025-01-08 00:50:54.346519] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4076, avg_group_log_size=370) [2025-01-08 00:50:54.346549] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4076) [2025-01-08 00:50:54.346566] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:54.357589] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:54.358970] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=66, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:54.360000] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=34] gc stale ls task succ [2025-01-08 00:50:54.371730] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690D66-0-0] [lt=22] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.796639385223388672e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=898, Throttle Trigger(MB)=538, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.733554252408669072e+01) [2025-01-08 00:50:54.380506] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC837E-0-0] [lt=36] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=20, avg_log_submit_to_flush_time=2009, avg_log_submit_to_slide_time=20879) [2025-01-08 00:50:54.380660] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:54.380710] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] succ to submit log task to replay service(to_submit_lsn={lsn:2179594}, to_submit_scn={val:1736326253988817003, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326254380477, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2179660}, next_to_submit_scn_:{val:1736326253988817004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2179538}, end_lsn:{lsn:2179660}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2179660}, readable_log_tail:{lsn:2179660}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64925108, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326253988817003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326253988817003, v:0}, data_checksum:3348837161, flag:35258}}, init_mode_version:0, accumulate_checksum:3193773187, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326253988817003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17819, miss_cnt:0, cache_read_size:2179660, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:54.381157] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:50:54.389656] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=1704, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:54.389764] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=106] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=11261) [2025-01-08 00:50:54.389789] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=284, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:54.389830] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:54.390878] INFO [LIB] stat (utility.h:1210) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA382F-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=22, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:54.390903] INFO [LIB] stat (utility.h:1210) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA382F-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4184, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:54.390912] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA382F-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=102, avg_log_submit_to_flush_time=2020, avg_log_submit_to_slide_time=4082) [2025-01-08 00:50:54.391175] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=11261, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:54.391414] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=46] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326253707292003, v:0}, new_max_version={val:1736326253707292003, v:0}, last_min_version={val:1736326253487654003, v:0}, last_max_version={val:1736326253487654003, v:0}, total_time_us=5306, delta_us=684119) [2025-01-08 00:50:54.391498] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=80] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326253707292003, v:0}, min_version={val:1736326253707292003, v:0}, max_version={val:1736326253707292003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:54.391664] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=48] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:54.391707] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=37] temporary file module use no blocks [2025-01-08 00:50:54.391719] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:54.391775] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=54] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:54.391812] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=35] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193758305389, flush_io_finished_ret_:0, flush_io_finished_round_:1913, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:54.398656] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:54.398748] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=85] temporary file module use no blocks [2025-01-08 00:50:54.398759] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:54.398839] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=78] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:54.398855] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193758312367, flush_io_finished_ret_:0, flush_io_finished_round_:3741, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:54.410023] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:54.413990] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=28] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:54.425302] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=31] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:54.436133] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:219) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:50:54.436238] INFO [COMMON] handle (memory_dump.cpp:538) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=103] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:50:54.437380] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] update_check_range(min_check_version=371, max_check_version=372, global_version=373) [2025-01-08 00:50:54.459052] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:54.459303] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=109] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:54.461869] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:54.462777] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=217] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:54.468322] INFO handle (memory_dump.cpp:665) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=33] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 150 262 0 1 2 68 1 0 1 4 1 1 0 1 5 42 5 0 1 7 1 2 0 1 8 48 1 0 1 12 7 3 0 1 13 1 1 0 1 14 1 2 0 1 15 1 1 0 1 18 4 3 0 1 20 1 2 0 500 0 94 237 0 500 7 7 4 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 2 1 0 500 17 9 7 0 500 22 79 61 0 500 23 23 10 0 508 0 2 10 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 104 230 0 1001 2 49 1 0 1001 4 1 1 0 1001 5 22 5 0 1001 8 30 1 0 1001 12 3 2 0 1001 13 1 1 0 1001 14 1 1 0 1001 18 4 3 0 1001 20 2 2 0 1002 0 162 245 0 1002 2 59 1 0 1002 4 2 1 0 1002 5 49 6 0 1002 7 11 2 0 1002 8 46 1 0 1002 12 2 3 0 1002 13 1 1 0 1002 14 1 1 0 1002 16 2 2 0 1002 18 9 3 0 1002 20 1 2 0 cost_time: 30972 [2025-01-08 00:50:54.468363] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant: 1, limit: 2,147,483,648 hold: 1,036,496,896 cache_hold: 213,909,504 cache_used: 213,909,504 cache_item_count: 102 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 443,588,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 142,606,336 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 93,765,632 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 100,663,296 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 14,680,064 limit= 429,496,720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [2025-01-08 00:50:54.468715] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 443,588,608 used= 339,551,296 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,211,520 used= 31,196,160 count= 15 avg_used= 2,079,744 block_cnt= 15 chunk_cnt= 15 mod=CACHE_MAP_NODE [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 22,437,888 used= 22,292,032 count= 22 avg_used= 1,013,274 block_cnt= 22 chunk_cnt= 16 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,196,912 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 37 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,268,096 used= 11,228,160 count= 4 avg_used= 2,807,040 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,618,560 used= 7,541,760 count= 10 avg_used= 754,176 block_cnt= 10 chunk_cnt= 8 mod=ResourceGroup [MEMORY] hold= 7,004,160 used= 6,012,928 count= 121 avg_used= 49,693 block_cnt= 121 chunk_cnt= 23 mod=[T]char [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMemCU [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,416,064 used= 2,993,504 count= 139 avg_used= 21,536 block_cnt= 139 chunk_cnt= 28 mod=DiagnosticInfo [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,768,896 used= 2,720,000 count= 13 avg_used= 209,230 block_cnt= 13 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 2,722,368 used= 2,607,696 count= 574 avg_used= 4,543 block_cnt= 558 chunk_cnt= 29 mod=WorkerMap [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,640 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 272 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,394,368 used= 1,372,072 count= 75 avg_used= 18,294 block_cnt= 75 chunk_cnt= 11 mod=TabletMap [MEMORY] hold= 1,374,720 used= 1,308,288 count= 36 avg_used= 36,341 block_cnt= 36 chunk_cnt= 14 mod=CommSysVarFac [MEMORY] hold= 1,327,104 used= 1,322,784 count= 9 avg_used= 146,976 block_cnt= 9 chunk_cnt= 8 mod=SQLSessionInfo [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 943,936 used= 920,576 count= 100 avg_used= 9,205 block_cnt= 100 chunk_cnt= 17 mod=ParamCharsType [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 884,736 used= 434,592 count= 54 avg_used= 8,048 block_cnt= 54 chunk_cnt= 17 mod=[T]ObPerfEventR [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 497,280 used= 477,120 count= 105 avg_used= 4,544 block_cnt= 105 chunk_cnt= 24 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 262,144 used= 261,120 count= 4 avg_used= 65,280 block_cnt= 4 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 262,144 used= 150,272 count= 16 avg_used= 9,392 block_cnt= 16 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 203,840 used= 194,432 count= 49 avg_used= 3,968 block_cnt= 49 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 12 mod=SqlSession [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 13 mod=LinearHashMapNo [MEMORY] hold= 155,024 used= 2,177 count= 1,920 avg_used= 1 block_cnt= 19 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 154,432 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 13 mod=QueryAllocator [MEMORY] hold= 147,456 used= 131,072 count= 2 avg_used= 65,536 block_cnt= 2 chunk_cnt= 2 mod=MinorMergeMgr [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 105,664 used= 103,168 count= 13 avg_used= 7,936 block_cnt= 13 chunk_cnt= 10 mod=SqlSessiVarMap [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 7 mod=PlanVaIdx [MEMORY] hold= 91,680 used= 23,840 count= 337 avg_used= 70 block_cnt= 75 chunk_cnt= 25 mod=Coro [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 5 mod=HashNodeLCSta [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 54,720 used= 51,840 count= 15 avg_used= 3,456 block_cnt= 12 chunk_cnt= 11 mod=CACHE_INST [MEMORY] hold= 50,144 used= 38,976 count= 58 avg_used= 672 block_cnt= 57 chunk_cnt= 18 mod=[T]ObWarningBuf [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 38,480 used= 26,880 count= 60 avg_used= 448 block_cnt= 30 chunk_cnt= 10 mod=OMT_Worker [MEMORY] hold= 37,328 used= 3,960 count= 165 avg_used= 24 block_cnt= 65 chunk_cnt= 26 mod=[T]MemoryContex [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PartCtx [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,640 used= 15,872 count= 4 avg_used= 3,968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnMemNode [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,640 used= 6,320 count= 12 avg_used= 526 block_cnt= 11 chunk_cnt= 8 mod=SqlSessiQuerSql [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=SqlSessionSbloc [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 7,504 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 6 mod=PackStateMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 6 mod=SequenceIdMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 8 mod=ContextsMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 6 mod=SequenceMap [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,848 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 5 mod=DICollector [MEMORY] hold= 4,864 used= 4,288 count= 3 avg_used= 1,429 block_cnt= 2 chunk_cnt= 2 mod=NegParamIdex [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3,520 used= 2,304 count= 6 avg_used= 384 block_cnt= 6 chunk_cnt= 6 mod=TableLockNode [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 2 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 320 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSIDDLVar [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 339,551,296 used= 331,990,522 count= 31,575 avg_used= 10,514 mod=SUMMARY [2025-01-08 00:50:54.468773] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=45] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 142,606,336 used= 141,492,224 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 block_cnt= 68 chunk_cnt= 68 mod=Memstore [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:50:54.468793] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:50:54.468814] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 93,765,632 used= 88,647,552 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 82,354,688 used= 74,462,706 count= 2,843 avg_used= 26,191 block_cnt= 2,843 chunk_cnt= 42 mod=SqlPhyPlan [MEMORY] hold= 4,643,904 used= 4,411,184 count= 306 avg_used= 14,415 block_cnt= 306 chunk_cnt= 40 mod=SqlPlanCache [MEMORY] hold= 1,305,600 used= 1,190,400 count= 600 avg_used= 1,984 block_cnt= 200 chunk_cnt= 39 mod=CreateContext [MEMORY] hold= 341,376 used= 333,312 count= 42 avg_used= 7,936 block_cnt= 42 chunk_cnt= 11 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 88,647,552 used= 80,399,202 count= 3,793 avg_used= 21,196 mod=SUMMARY [2025-01-08 00:50:54.468834] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,368 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,272 used= 58,751 count= 259 avg_used= 226 block_cnt= 21 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,368 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-01-08 00:50:54.468848] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 100,663,296 used= 95,993,856 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 95,993,856 used= 95,815,296 count= 186 avg_used= 515,136 block_cnt= 186 chunk_cnt= 48 mod=CoStack [MEMORY] hold= 95,993,856 used= 95,815,296 count= 186 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:50:54.468869] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 213,909,504 used= 213,909,504 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 mod=SUMMARY [2025-01-08 00:50:54.468886] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 14,680,064 used= 12,713,920 limit= 429,496,720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,036,352 used= 7,149,419 count= 109 avg_used= 65,591 block_cnt= 109 chunk_cnt= 5 mod=L_TabletPool [MEMORY] hold= 4,587,520 used= 4,569,600 count= 70 avg_used= 65,280 block_cnt= 70 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 12,713,920 used= 11,803,787 count= 181 avg_used= 65,214 mod=SUMMARY [2025-01-08 00:50:54.468900] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,584,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 block_cnt= 195 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:50:54.468915] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 16,256 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ExtInfoLog [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:50:54.468929] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 33,216 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 mod=SUMMARY [2025-01-08 00:50:54.468957] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 13,993,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 9,473,984 used= 9,445,184 count= 3 avg_used= 3,148,394 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 4,511,040 used= 4,404,480 count= 555 avg_used= 7,936 block_cnt= 555 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 13,993,152 used= 13,857,600 count= 559 avg_used= 24,789 mod=SUMMARY [2025-01-08 00:50:54.468976] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 73,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 73,152 used= 71,424 count= 9 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:50:54.470449] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=1437] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 631,246,848 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 232,894,464 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 24,260,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 170,676,224 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98,557,952 limit= 9,223,372,036,854,775,807 [2025-01-08 00:50:54.470983] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 232,894,464 used= 185,976,416 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,397,376 used= 7,340,256 count= 7 avg_used= 1,048,608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7,259,504 used= 6,324,350 count= 1,032 avg_used= 6,128 block_cnt= 540 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,783,552 used= 5,752,960 count= 6 avg_used= 958,826 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 16 mod=Iteratorget_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:54.564446] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:81) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EC9-0-0] [lt=40] MergeLoopTask(cost_ts=68683) [2025-01-08 00:50:54.564621] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:54.566871] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8383-0-0] [lt=44] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:54.566893] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8383-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4929, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:54.567686] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FCB-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:54.567705] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FCB-0-0] [lt=18] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5574, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:54.567714] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FCB-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=106, avg_log_submit_to_flush_time=2953, avg_log_submit_to_slide_time=5467) [2025-01-08 00:50:54.570728] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCF-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:54.570759] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCF-0-0] [lt=25] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=34) [2025-01-08 00:50:54.573080] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690D69-0-0] [lt=48] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=131) [2025-01-08 00:50:54.574346] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=51] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:54.579381] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A444D-0-0] [lt=68] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:54.579446] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A444D-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:54.579459] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A444D-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21139, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:54.579467] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A444D-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17908, log_task={header:{begin_lsn:{lsn:2179660}, end_lsn:{lsn:2179782}, log_id:17908, min_scn:{val:1736326253988817004, v:0}, max_scn:{val:1736326253988817004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2179538}, prev_proposal_id:1, committed_end_lsn:{lsn:2179660}, data_checksum:1715074492, accum_checksum:2399902202, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326254478433, freeze_ts:1736326254478433, submit_ts:1736326254478442, flushed_ts:1736326254480259, gen_to_freeze cost time:0, gen_to_submit cost time:9, submit_to_flush cost time:1817}, fs_cb_begin_ts=1736326254579436, log_life_time=101003) [2025-01-08 00:50:54.581414] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=96] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:50:54.583813] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:54.590808] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=36] [APPLY STAT CB APPEND COST TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=134, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:50:54.590849] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=41] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=4898, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:50:54.590862] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:50:54.590870] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=-1114, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:50:54.590877] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=3918, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:50:54.619281] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:54.649271] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:54.659156] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5187579415, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:54.659263] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:54.659278] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d4205cf0) [2025-01-08 00:50:54.659288] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5187579415, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:54.660683] INFO [LIB] stat (utility.h:1210) [58346][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3713-0-0] [lt=36] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:54.660740] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58346][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3713-0-0] [lt=42] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:54.661451] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:54.662498] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58324][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=43] worker thread exit(tenant_->id()=1002, workers_.get_size()=6) [2025-01-08 00:50:54.663333] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FCD-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:54.663466] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FCD-0-0] [lt=132] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23293, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:54.663871] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1977, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:54.663925] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=850, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:54.666090] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=100, this=0x2ad934926280, extra_info="") [2025-01-08 00:50:54.666116] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=24] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=5173, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:50:54.666125] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:50:54.666132] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=-795, this=0x2ad934926910, extra_info="") [2025-01-08 00:50:54.666140] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=4478, this=0x2ad934926b40, extra_info="") [2025-01-08 00:50:54.666873] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=203] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1765, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:54.699467] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A5-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326254694405, keepalive_ts:-1}) [2025-01-08 00:50:54.699520] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A5-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326254694405, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:54.699811] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A6-0-0] [lt=17] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326254694489, keepalive_ts:-1}) [2025-01-08 00:50:54.699834] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A6-0-0] [lt=23] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326254694489, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:54.709667] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=105] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=11850, avg_group_log_size=623) [2025-01-08 00:50:54.709712] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11850) [2025-01-08 00:50:54.719400] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:54.749364] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1451, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:54.749407] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3652) [2025-01-08 00:50:54.749520] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:54.749500] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=298, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:54.751159] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6764-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:54.751256] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6764-0-0] [lt=95] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3801, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:54.751270] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6764-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=106, avg_log_submit_to_flush_time=1956, avg_log_submit_to_slide_time=3694) [2025-01-08 00:50:54.751435] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB APPEND COST TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=134, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:50:54.751454] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=19] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=5003, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:50:54.751463] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:50:54.751532] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=68] [APPLY STAT CB EXECUTE TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=-1015, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:50:54.751541] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] [APPLY STAT CB TOTAL TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=4122, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:50:54.751650] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3652, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:54.758698] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:54.767045] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:54.772129] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=134][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:50:54.772361] WDIAG read_multi_row (ob_table_access_helper.h:238) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=112][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:50:54.774224] INFO delete_row (ob_table_access_helper.h:339) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=46] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326254772409) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326254772409) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:50:54.774660] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1736326254572496000, v:0}, base_version={val:1736326254371546000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326254572496000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326254774642, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326254572496000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326254707695, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326254572496000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326254842197, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:50:54.778562] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=36] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:54.778784] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=82] ====== tenant freeze timer task ====== [2025-01-08 00:50:54.781777] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=60] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:50:54.784929] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085143-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:54.785048] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085145-0-0] [lt=24] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:54.793510] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D00E-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:54.801473] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0EF-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326254796873, keepalive_ts:-1}) [2025-01-08 00:50:54.801508] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0EF-0-0] [lt=34] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326254796873, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:54.811585] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=50] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:54.819530] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:54.857633] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=46] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1003}, {id:1}, {id:1001}]) [2025-01-08 00:50:54.862628] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=47] PNIO [ratelimit] time: 1736326254862625, bytes: 867407234, bw: 0.191156 MB/s, add_ts: 1005991, add_bytes: 201642 [2025-01-08 00:50:54.877535] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4453-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:50:54.877564] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4453-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:50:54.879300] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18646361}], scn_array=[cnt:1, {val:1736326254413263004, v:0}], curr_size=122, accum_size=1643, time_cost=10596) [2025-01-08 00:50:54.879355] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:50:54.880689] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1545, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:54.880758] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=68] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:54.880775] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=156, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:54.880838] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:54.890525] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=145] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:50:54.891408] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=36] ====== checkpoint timer task ====== [2025-01-08 00:50:54.892635] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=22] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:54.892677] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=42] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:54.892777] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:54.892947] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=165] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:50:54.893012] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=58] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:54.893028] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1736326252704413001, v:0}) [2025-01-08 00:50:54.893037] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:50:54.893045] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:50:54.893059] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:54.893077] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:54.893091] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:54.893110] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:54.893130] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:54.893140] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:50:54.895463] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=33] ====== check clog disk timer task ====== [2025-01-08 00:50:54.895542] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=73] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:50:54.896373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=44] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:54.898426] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:54.898479] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:54.898503] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:54.898567] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=68] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:50:54.914379] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=185] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:54.914469] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58353][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=25] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:50:54.915551] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B9D-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11148) [2025-01-08 00:50:54.922666] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:54.923339] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:54.926940] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:195) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C86-0-0] [lt=47][errcode=-4018] table_operator_ get_source failed(ret=-4018) [2025-01-08 00:50:54.929717] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=38] ====== check clog disk timer task ====== [2025-01-08 00:50:54.929744] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:54.931046] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C86-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:54.933112] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C86-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:54.938242] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:32338, max_lsn:43048189, max_scn:{val:1736326254911082000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:43048189}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:1105149}}, last_submit_lsn:{lsn:43047600}, last_submit_end_lsn:{lsn:43048189}, last_submit_log_id:32338, last_submit_log_pid:1, max_flushed_lsn:{lsn:43047600}, max_flushed_end_lsn:{lsn:43048189}, max_flushed_log_pid:1, committed_end_lsn:{lsn:43048189}, last_slide_log_id:32338, last_slide_scn:{val:1736326254911082000, v:0}, last_slide_lsn:{lsn:43047600}, last_slide_end_lsn:{lsn:43048189}, last_slide_log_pid:1, last_slide_log_accum_checksum:3034415255, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad9083ee530}, [StateMgr]={this:0x2ad9083f2830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:32339, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324348477669, palf_role_change_cb:0x2ad9083f92f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad9083efc70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324348678844, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/meta", dir_fd:172, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad9083f4a38, flashback_version:0}, log_storage:{log_tail:{lsn:43048189}, readable_log_tail:{lsn:43048189}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:24056579, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324348477523, this:0x2ad9083f2f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.138:2882";"192.168.159.139:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324348478003, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9083f29b0}) [2025-01-08 00:50:54.938630] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=380] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326254922535, last_flushed_end_lsn:{lsn:43048189}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326254915902, last_flushed_end_lsn:{lsn:43048189}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326254915542, last_flushed_end_lsn:{lsn:43048189}}]) [2025-01-08 00:50:54.938717] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=82] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=32339, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326254922535, last_flushed_end_lsn:{lsn:43048189}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326254915902, last_flushed_end_lsn:{lsn:43048189}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326254915542, last_flushed_end_lsn:{lsn:43048189}}]) [2025-01-08 00:50:54.949286] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=10] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:50:54.949655] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=105, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:54.952905] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=32] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12815347712, fast_delete_log_mode=false) [2025-01-08 00:50:54.959169] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=60] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735339520, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:54.962632] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:54.963024] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:54.963308] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:54.965078] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:54.965131] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:54.968901] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1643) [2025-01-08 00:50:54.970863] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1765, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:54.970889] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:54.970906] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=157, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:54.971102] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:54.977920] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4455-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17912, log_task={header:{begin_lsn:{lsn:2180148}, end_lsn:{lsn:2180270}, log_id:17912, min_scn:{val:1736326254487295003, v:0}, max_scn:{val:1736326254487295003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2180026}, prev_proposal_id:1, committed_end_lsn:{lsn:2180148}, data_checksum:1715074492, accum_checksum:3981895980, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326254877497, freeze_ts:1736326254877498, submit_ts:1736326254877510, flushed_ts:1736326254880799, gen_to_freeze cost time:1, gen_to_submit cost time:13, submit_to_flush cost time:3289}, fs_cb_begin_ts=1736326254977909, log_life_time=100412) [2025-01-08 00:50:54.978273] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=97, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:54.978296] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:54.981122] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17825, miss_cnt:0, cache_read_size:2180392, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:54.981946] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:50:55.010629] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D30-0-0] [lt=40] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:55.025009] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594F-0-0] [lt=18] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12274, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:55.025904] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:55.025932] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.026780] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594F-0-0] [lt=22] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12274, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326255026756, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326255026756, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326255026756, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:55.026918] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594F-0-0] [lt=136] success to process renew task(task={tenant_id:1, table_id:12274, add_timestamp:1736326255024861}, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326255026756, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326255026756, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326255026756, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:55.029571] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:55.029662] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:10, bw:40493, [delay/us]:prepare:8, schedule:112, submit:76, rt:1516, total:1712) [2025-01-08 00:50:55.029680] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:55.029734] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:10, avg_bw:40495, [delay/us]: prepare:8, schedule:112, submit:76, device:1516, total:1712) [2025-01-08 00:50:55.029747] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:55.029786] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:72893, [delay/us]:prepare:9, schedule:795, submit:127, rt:1203, total:2135) [2025-01-08 00:50:55.029796] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:55.029950] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=140] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:72886, [delay/us]: prepare:9, schedule:795, submit:127, device:1203, total:2135) [2025-01-08 00:50:55.029963] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:55.029987] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16197, [delay/us]:prepare:20, schedule:31, submit:1, rt:928, total:981) [2025-01-08 00:50:55.030038] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:170070, [delay/us]:prepare:7, schedule:144, submit:261, rt:1311, total:1724) [2025-01-08 00:50:55.030049] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:55.030080] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16196, [delay/us]: prepare:20, schedule:31, submit:1, device:928, total:981) [2025-01-08 00:50:55.030096] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:170065, [delay/us]: prepare:7, schedule:144, submit:261, device:1311, total:1724) [2025-01-08 00:50:55.030106] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:55.030126] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4049, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:300, total:308) [2025-01-08 00:50:55.030150] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:55.030176] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4049, [delay/us]: prepare:0, schedule:0, submit:600000000, device:300, total:308) [2025-01-08 00:50:55.030194] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:55.030209] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:55.030287] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:55.030298] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:55.030306] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=199kB/s, net_out=223kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:55.054579] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:50:55.054673] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=86] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:50:55.054691] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=14] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:50:55.054713] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=20] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:55.058919] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=25] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:50:55.061162] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=24] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:50:55.062725] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=107] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:50:55.062773] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=45] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:50:55.064708] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084875-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:50:55.064802] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084875-0-0] [lt=48] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:50:55.064819] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084875-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:50:55.065442] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:50:55.065474] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:50:55.068104] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC371B-0-0] [lt=101] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:55.069523] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:55.070608] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:50:55.071617] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3243, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:55.071823] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=200] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2041, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:55.071915] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:55.072002] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=22] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:50:55.074119] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=37] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:50:55.074585] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=19] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:55.076309] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=48] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:50:55.076336] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=25] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:50:55.078322] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084875-0-0] [lt=9] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:50:55.078474] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084875-0-0] [lt=19] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:50:55.078494] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084875-0-0] [lt=17] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:50:55.082431] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=12] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:50:55.084127] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=18] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:50:55.085591] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:55.086349] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=18] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:50:55.086374] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=24] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:50:55.087763] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084875-0-0] [lt=35] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:50:55.087851] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084875-0-0] [lt=37] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:50:55.087889] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084875-0-0] [lt=35] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:50:55.087914] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=15] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:50:55.088554] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=637] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:55.088607] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=45] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:50:55.088619] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:55.088639] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084875-0-0] [lt=18] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:50:55.088658] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084875-0-0] [lt=18] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:50:55.088669] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=7] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:50:55.088675] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=7] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:55.088691] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084875-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:50:55.088703] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084875-0-0] [lt=11] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:50:55.088712] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=5] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:50:55.088718] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:55.088734] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084875-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:50:55.088746] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084875-0-0] [lt=12] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:50:55.090100] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:50:55.090152] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=49] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:50:55.090164] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=11] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:50:55.090269] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=72] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:50:55.090282] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=13] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:50:55.090290] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=7] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:50:55.090300] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:50:55.090315] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084875-0-0] [lt=13] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:50:55.120531] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=81, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:55.123986] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:55.126026] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.149515] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:55.153103] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3539) [2025-01-08 00:50:55.159593] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=27] ====== check clog disk timer task ====== [2025-01-08 00:50:55.159688] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=82] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:55.160353] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=136] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:55.162506] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=98] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=440398, clean_num=31457) [2025-01-08 00:50:55.162626] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C293-0-0] [lt=110] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:55.162673] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C293-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326254487295004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326254487295004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:55.175295] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=102] no need to update min/max version(new_version={val:1736326254775798000, v:0}, min_version={val:1736326254775798000, v:0}, max_version={val:1736326254775798000, v:0}) [2025-01-08 00:50:55.176478] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC377-0-0] [lt=92][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:55.181998] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=34] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:288K/68:322512 disk:288K/68:322512 t1m2:36K/9:82014 t1m2g0:36K/9:82014 t1002m2:176K/43:218739 t1002m2g0:176K/43:218739 t1001m2:76K/16:736683 t1001m2g0:76K/16:736683 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:55.182101] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:55.193050] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1736326254917395001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2147479}, palf_committed_end_scn_:{val:1736326254917395002, v:0}, last_check_scn_:{val:1736326254917395002, v:0}, max_applied_cb_scn_:{val:1736326254917395001, v:0}}) [2025-01-08 00:50:55.193094] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=41] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2147479}, readable_log_tail:{lsn:2147479}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64957289, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:55.207904] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE7E5-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326255207433000, start_id=1736326255207433000, end_id=1736326255207433001) [2025-01-08 00:50:55.215673] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:50:55.219069] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=82] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8572, offset=2431666}, end_cursor:ObLogCursor{file_id=1, log_id=8573, offset=2431971}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946e0050, len:1358, log_data_len:305, seq:8572, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:55.219534] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E40-0-0] [lt=98] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:55.219590] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E40-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326254917395002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326254917395002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:55.223596] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:55.223639] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:55.224937] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87028-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:55.225007] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:55.225021] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:55.226150] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=71][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.229401] INFO [LIB] stat (utility.h:1210) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E40-0-0] [lt=21] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:55.233543] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8573, offset=2431971}, end_cursor:ObLogCursor{file_id=1, log_id=8574, offset=2432268}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946e4050, len:1053, log_data_len:297, seq:8573, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:55.234096] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E40-0-0] [lt=50] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:55.234318] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E40-0-0] [lt=204] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326254917395002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326254917395002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:55.242295] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=53] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8574, offset=2432268}, end_cursor:ObLogCursor{file_id=1, log_id=8575, offset=2432565}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946e8050, len:756, log_data_len:297, seq:8574, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:55.243610] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=96] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8575, offset=2432565}, end_cursor:ObLogCursor{file_id=1, log_id=8576, offset=2432862}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946ec050, len:459, log_data_len:297, seq:8575, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:55.243875] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17740, miss_cnt:0, cache_read_size:2170180, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:55.246500] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=63] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:50:55.246576] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=70] start lib cache evict(tenant_id=1, mem_hold=93765632, mem_limit=107374180, cache_obj_num=300, cache_node_num=300) [2025-01-08 00:50:55.246589] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=93765632, mem_limit=107374180, cache_obj_num=300, cache_node_num=300) [2025-01-08 00:50:55.246599] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:55.249421] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=300, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}]) [2025-01-08 00:50:55.250224] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=800] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:55.250726] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=51] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:55.250748] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=21] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:55.250797] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E3-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:55.250863] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E3-0-0] [lt=62] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:55.264362] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FDF-0-0] [lt=27] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=15, avg_log_submit_to_flush_time=2238, avg_log_submit_to_slide_time=22892) [2025-01-08 00:50:55.264442] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=81] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:55.278392] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=40] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:55.278760] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A445B-0-0] [lt=54][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17915, log_task={header:{begin_lsn:{lsn:2180505}, end_lsn:{lsn:2180627}, log_id:17915, min_scn:{val:1736326254991003001, v:0}, max_scn:{val:1736326254991003001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2180392}, prev_proposal_id:1, committed_end_lsn:{lsn:2180505}, data_checksum:1715074492, accum_checksum:549432081, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326255177411, freeze_ts:1736326255177411, submit_ts:1736326255177421, flushed_ts:1736326255178839, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:1418}, fs_cb_begin_ts=1736326255278750, log_life_time=101339) [2025-01-08 00:50:55.281884] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:50:55.282236] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854B1-0-0] [lt=23] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:50:55.282258] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:55.282285] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-01-08 00:50:55.282284] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854B1-0-0] [lt=26] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:50:55.282295] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:55.282308] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:55.282321] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193759196036, flush_io_finished_ret_:0, flush_io_finished_round_:1918, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:55.290272] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9D-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:55.290325] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9D-0-0] [lt=50] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:55.303398] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185412-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:55.314501] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF26-0-0] [lt=25] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326255309888, keepalive_ts:-1}) [2025-01-08 00:50:55.314550] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF26-0-0] [lt=48] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326255309888, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:55.320700] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.320742] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.326271] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.331467] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB56-0-0] [lt=170][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193760244999, cache_obj->added_lc()=false, cache_obj->get_object_id()=13560, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:55.336061] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=43] current monitor number(seq_=-1) [2025-01-08 00:50:55.337173] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] Garbage Collector is running(seq_=376, gc_interval=10000000) [2025-01-08 00:50:55.337279] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=88] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:50:55.337301] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=13] gc_check_member_list_ cost time(ret=0, time_us=40) [2025-01-08 00:50:55.337314] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:50:55.337326] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:55.337339] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:55.337349] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=11) [2025-01-08 00:50:55.337357] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] ObSafeDestroyHandler start process [2025-01-08 00:50:55.342130] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1609BCBC-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1736326255340374000, start_id=1736326255340374000, end_id=1736326255340374001) [2025-01-08 00:50:55.360268] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:50:55.366113] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:55.366153] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=76, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:55.378803] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58333][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A445D-0-0] [lt=46][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17916, log_task={header:{begin_lsn:{lsn:2180627}, end_lsn:{lsn:2180749}, log_id:17916, min_scn:{val:1736326254991003002, v:0}, max_scn:{val:1736326254991003002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2180505}, prev_proposal_id:1, committed_end_lsn:{lsn:2180627}, data_checksum:1715074492, accum_checksum:477872007, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326255278697, freeze_ts:1736326255278697, submit_ts:1736326255278739, flushed_ts:1736326255280966, gen_to_freeze cost time:0, gen_to_submit cost time:42, submit_to_flush cost time:2227}, fs_cb_begin_ts=1736326255378791, log_life_time=100094) [2025-01-08 00:50:55.378953] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB024-0-0] [lt=86] get number(ret=0, service_type_=0, range=1, base_id=1736326255378620000, start_id=1736326255378620000, end_id=1736326255378620001) [2025-01-08 00:50:55.382022] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58333][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D66D-0-0] [lt=58] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=15, avg_log_submit_to_flush_time=2830, avg_log_submit_to_slide_time=70652) [2025-01-08 00:50:55.382952] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=68] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:55.384242] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=11] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1100960, replace_num=15728) [2025-01-08 00:50:55.384281] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:55.390602] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AC2C-0-0] [lt=137] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.718940544128417969e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=385, Throttle Trigger(MB)=231, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.906445290688600380e+02) [2025-01-08 00:50:55.391577] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=32] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:55.391625] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=39] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:55.391641] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=85) [2025-01-08 00:50:55.392039] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:55.392084] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=37] temporary file module use no blocks [2025-01-08 00:50:55.392097] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:55.392110] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:55.392123] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193759305686, flush_io_finished_ret_:0, flush_io_finished_round_:1914, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:55.392564] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=128] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=2113, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:55.392591] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9274) [2025-01-08 00:50:55.392632] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=154, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:55.392703] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=36] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:55.393653] INFO [LIB] stat (utility.h:1210) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BA5-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=27, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:55.393677] INFO [LIB] stat (utility.h:1210) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BA5-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=6151, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:55.393687] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BA5-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=120, avg_log_submit_to_flush_time=4170, avg_log_submit_to_slide_time=6031) [2025-01-08 00:50:55.395008] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=96] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9274, hit_cnt=14, read_cnt=14, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:55.395803] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=59] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326254705945003, v:0}, new_max_version={val:1736326254705945003, v:0}, last_min_version={val:1736326254413263004, v:0}, last_max_version={val:1736326254413263004, v:0}, total_time_us=6718, delta_us=689855) [2025-01-08 00:50:55.395848] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=42] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326254705945003, v:0}, min_version={val:1736326254705945003, v:0}, max_version={val:1736326254705945003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:55.404557] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:55.404757] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=187] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:55.404833] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=72] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:50:55.404842] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:50:55.404849] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:50:55.404857] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=92) [2025-01-08 00:50:55.415931] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1609BCBC-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1736326255415498000, start_id=1736326255415498000, end_id=1736326255415498001) [2025-01-08 00:50:55.423352] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1736326255421557000, start_id=1736326255421557000, end_id=1736326255421557001) [2025-01-08 00:50:55.423387] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=29] ts response task statistics(alloc_count=59526, free_count=29762) [2025-01-08 00:50:55.423407] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=566, get_gts_with_stc_cnt=77, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=53, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:55.423502] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=77] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=506, get_gts_with_stc_cnt=75, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=73, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:55.426710] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.452270] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=3661, avg_group_log_size=332) [2025-01-08 00:50:55.452325] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3661) [2025-01-08 00:50:55.452347] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:55.459252] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.459313] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.463856] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=658198528, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=415543296, wash_ratio=6, tenant_id=1001) [2025-01-08 00:50:55.463953] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=59] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:50:55.463976] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=20] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2801, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:55.464037] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1112293376, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2108932096, wash_ratio=6, tenant_id=1002) [2025-01-08 00:50:55.464090] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:50:55.464104] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:55.464186] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:55.465085] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:55.467358] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=23] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:55.472295] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D160BB024-0-0] [lt=73] get number(ret=0, service_type_=0, range=1, base_id=1736326255471336000, start_id=1736326255471336000, end_id=1736326255471336001) [2025-01-08 00:50:55.472628] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=40] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:55.474807] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=106] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:55.479276] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:50:55.479304] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70075580, tenant_id=1001) [2025-01-08 00:50:55.479338] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=25] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:55.493701] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=81] ====== tenant freeze timer task ====== [2025-01-08 00:50:55.519274] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.521678] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=87] PNIO [ratelimit] time: 1736326255521675, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007509, add_bytes: 0 [2025-01-08 00:50:55.526827] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.542264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.542876] INFO pn_ratelimit (group.c:653) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:50:55.544312] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.544386] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E9-0-0] [lt=34] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:55.544469] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E9-0-0] [lt=64] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326255207433001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326255207433001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:55.559817] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5E9-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326255558594000, start_id=1736326255558594000, end_id=1736326255558594001) [2025-01-08 00:50:55.562998] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=43058452, this={dio_aligned_buf:{buf_write_offset:2050, buf_padding_size:2046, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:45276, truncate_used_ts:13113}, log_block_size:67108864, io_fd:{first_id:-1, second_id:160, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=43057152, buf_len=750, total_write_size=100179, total_write_size_after_dio=819200, ob_pwrite_used_ts=438906, count=176, each_pwrite_cost=2493) [2025-01-08 00:50:55.564111] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:55.566697] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=83] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:55.566974] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=145] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=68, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:55.567003] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:55.567020] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:55.567034] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:55.569379] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:55.569824] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83A0-0-0] [lt=28] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:55.569847] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83A0-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5980, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:55.569856] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83A0-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=119, avg_log_submit_to_flush_time=4587, avg_log_submit_to_slide_time=5861) [2025-01-08 00:50:55.571435] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:50:55.573120] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83A1-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=24, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:55.573156] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83A1-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6074, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:55.574902] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:55.584436] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:50:55.585763] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:55.605450] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=35] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:50:55.627053] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=89][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.649850] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=106] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:55.660434] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3725-0-0] [lt=16][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17832, log_task={header:{begin_lsn:{lsn:2170424}, end_lsn:{lsn:2170546}, log_id:17832, min_scn:{val:1736326255458080000, v:0}, max_scn:{val:1736326255458080000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2170302}, prev_proposal_id:1, committed_end_lsn:{lsn:2170424}, data_checksum:940009244, accum_checksum:1789202764, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326255465441, freeze_ts:1736326255465442, submit_ts:1736326255465453, flushed_ts:1736326255471440, gen_to_freeze cost time:1, gen_to_submit cost time:12, submit_to_flush cost time:5987}, fs_cb_begin_ts=1736326255660422, log_life_time=194981) [2025-01-08 00:50:55.666183] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D670-0-0] [lt=76] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=26, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:55.666312] INFO [LIB] stat (utility.h:1210) [58333][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D670-0-0] [lt=127] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=46640, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:55.669002] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:55.669378] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854B3-0-0] [lt=13] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:50:55.669424] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854B3-0-0] [lt=37] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:50:55.669433] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=72] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5188828512, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:55.669462] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:55.669474] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad90857dd30) [2025-01-08 00:50:55.669555] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=79] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5188828512, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:55.671209] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3643, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:55.671256] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=717, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:55.671659] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58333][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=11] worker thread exit(tenant_->id()=1002, workers_.get_size()=5) [2025-01-08 00:50:55.679698] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4461-0-0] [lt=30] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:55.679765] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4461-0-0] [lt=31] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:55.679779] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4461-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=80461, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:55.679903] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4461-0-0] [lt=122][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17918, log_task={header:{begin_lsn:{lsn:2180871}, end_lsn:{lsn:2180993}, log_id:17918, min_scn:{val:1736326255433262000, v:0}, max_scn:{val:1736326255433262000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2180749}, prev_proposal_id:1, committed_end_lsn:{lsn:2180871}, data_checksum:1715074492, accum_checksum:2400776448, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326255480827, freeze_ts:1736326255480827, submit_ts:1736326255480834, flushed_ts:1736326255483649, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:2815}, fs_cb_begin_ts=1736326255679756, log_life_time=198929) [2025-01-08 00:50:55.714260] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=80] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:50:55.721514] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9789, avg_group_log_size=652) [2025-01-08 00:50:55.721548] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9789) [2025-01-08 00:50:55.727158] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.760470] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3727-0-0] [lt=58] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:55.760518] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3727-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:55.767083] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:55.767214] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:55.772685] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2033, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:55.778929] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:55.786330] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:50:55.792115] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D011-0-0] [lt=102][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:55.811837] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=50] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:55.827296] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.854503] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=2081, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:55.854552] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4988) [2025-01-08 00:50:55.854577] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=143, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:55.854613] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=45, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:55.855337] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6771-0-0] [lt=82] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:55.855360] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6771-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=4747, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:55.855370] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6771-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=115, avg_log_submit_to_flush_time=2336, avg_log_submit_to_slide_time=4632) [2025-01-08 00:50:55.855513] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4988, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:55.857831] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1003}, {id:1002}, {id:1}]) [2025-01-08 00:50:55.866139] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3782) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-01-08 00:50:55.866258] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3805) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=117] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-01-08 00:50:55.867325] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1736326255867320, bytes: 867715145, bw: 0.292275 MB/s, add_ts: 1004695, add_bytes: 307911 [2025-01-08 00:50:55.876840] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326255675469000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=201357, in_cluster_service=true, cluster_version={val:1736326255421557000, v:0}, min_cluster_version={val:1736326255421557000, v:0}, max_cluster_version={val:1736326255421557000, v:0}, get_cluster_version_err=0, cluster_version_delta=455269, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36305, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=510373813, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:55.880463] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4465-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1089, avg_group_log_size=121) [2025-01-08 00:50:55.880508] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4465-0-0] [lt=44] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1089) [2025-01-08 00:50:55.882612] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2089, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:55.882639] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1089) [2025-01-08 00:50:55.882657] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=744, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:55.882781] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=199] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=60, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:55.897315] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.897359] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.897384] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.897406] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:55.898640] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=182] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:55.920709] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:55.921173] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58341][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=11] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:50:55.923027] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=53] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9789) [2025-01-08 00:50:55.924087] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:55.927388] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=101][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:55.941918] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C87-0-0] [lt=109][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:55.943489] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C87-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:55.964803] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:55.965307] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:55.967685] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:55.969400] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:55.971301] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2033) [2025-01-08 00:50:55.971404] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1911) [2025-01-08 00:50:55.971465] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=12] Garbage Collector is running(seq_=192, gc_interval=10000000) [2025-01-08 00:50:55.971790] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=50] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:50:55.971839] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=39] gc_check_member_list_ cost time(ret=0, time_us=323) [2025-01-08 00:50:55.971857] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=12] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:50:55.974594] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=6] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:55.974688] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=42] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:55.974704] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=15] execute_gc cost time(ret=0, time_us=17) [2025-01-08 00:50:55.974713] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:50:55.975644] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:55.975734] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=36, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:55.979569] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=34] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=568, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:55.979594] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:55.982406] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17834, miss_cnt:0, cache_read_size:2181481, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:55.986632] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=93] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:50:56.027545] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.033316] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:56.033398] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48967, [delay/us]:prepare:8, schedule:468, submit:46, rt:1227, total:1751) [2025-01-08 00:50:56.033415] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:56.033504] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=70] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48967, [delay/us]: prepare:8, schedule:468, submit:46, device:1227, total:1751) [2025-01-08 00:50:56.033520] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:56.033561] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4915, iops:15, bw:73450, [delay/us]:prepare:7, schedule:145, submit:61, rt:1227, total:1442) [2025-01-08 00:50:56.033571] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:56.033683] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=96] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4915, avg_iops:15, avg_bw:73453, [delay/us]: prepare:7, schedule:145, submit:61, device:1227, total:1442) [2025-01-08 00:50:56.033697] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:56.033721] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16323, [delay/us]:prepare:6, schedule:66, submit:6, rt:1013, total:1092) [2025-01-08 00:50:56.033743] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4203, iops:38, bw:159149, [delay/us]:prepare:6, schedule:245, submit:410, rt:1363, total:2026) [2025-01-08 00:50:56.033834] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=86] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:56.033874] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16322, [delay/us]: prepare:6, schedule:66, submit:6, device:1013, total:1092) [2025-01-08 00:50:56.033891] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4204, avg_iops:38, avg_bw:159140, [delay/us]: prepare:6, schedule:245, submit:410, device:1363, total:2026) [2025-01-08 00:50:56.033900] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:56.033926] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4080, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:399, total:409) [2025-01-08 00:50:56.033951] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:56.033982] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4080, [delay/us]: prepare:0, schedule:0, submit:600000000, device:399, total:409) [2025-01-08 00:50:56.034000] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:56.034015] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:56.034027] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:56.034036] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:56.034044] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=304kB/s, net_out=262kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:56.042119] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB58-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193760955951, cache_obj->added_lc()=false, cache_obj->get_object_id()=13561, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:56.044161] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B8F-0-0] [lt=12] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:56.051900] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB5D-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1736326256051698000, start_id=1736326256051698000, end_id=1736326256051698001) [2025-01-08 00:50:56.052093] INFO [LIB] stat (utility.h:1210) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB5D-0-0] [lt=50] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=74, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:56.052143] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:149) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB5D-0-0] [lt=20] transaction rpc statistics(total_rt=12380, total_process=46, avg_rt=263) [2025-01-08 00:50:56.054831] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=66] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:50:56.061956] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1960, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:56.062005] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:56.062030] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=304, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:56.062102] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=65, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:56.069468] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1911, avg_group_log_size=191) [2025-01-08 00:50:56.069515] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1911) [2025-01-08 00:50:56.071693] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:56.072644] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=118] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:56.075544] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:56.086072] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:56.109109] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50506][T1002_L0_G0][T1002][YB42C0A89F8B-00062B2D1B5BBA52-0-0] [lt=1038] get number(ret=0, service_type_=0, range=1, base_id=1736326256104493000, start_id=1736326256104493000, end_id=1736326256104493001) [2025-01-08 00:50:56.115466] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2289, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:56.115499] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=212, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:56.115590] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=131] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=63, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:56.121913] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=78, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:56.127673] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.128087] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [46346][T1_ArcFetcher][T1][YB42C0A89F89-00062B2D1678458E-0-0] [lt=44] ObArchiveFetcher is running(thread_index=0) [2025-01-08 00:50:56.130791] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:50:56.130795] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58356][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:56.134319] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326252110999, last_first_register_time_us=1736324449183670, register_time_us=1736326252110999, curr_time_us=1736326256134316) [2025-01-08 00:50:56.134401] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=45] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326256134366, last_submit_register_req_time_us=1736326256134366, register_parent_reason=1) [2025-01-08 00:50:56.134420] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:50:56.134440] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1003, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:17837, max_lsn:2171156, max_scn:{val:1736326255707616003, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2171156}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2171034}, last_submit_end_lsn:{lsn:2171156}, last_submit_log_id:17837, last_submit_log_pid:1, max_flushed_lsn:{lsn:2171034}, max_flushed_end_lsn:{lsn:2171156}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2171156}, last_slide_log_id:17837, last_slide_scn:{val:1736326255707616003, v:0}, last_slide_lsn:{lsn:2171034}, last_slide_end_lsn:{lsn:2171156}, last_slide_log_pid:1, last_slide_log_accum_checksum:2100271640, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924fd6530}, [StateMgr]={this:0x2ad924fda830, self:"192.168.159.137:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.139:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:17838, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad924fe12f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326253112995, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326253112995, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924fd7c70}, [ModeMgr]={palf_id:1003, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436825152001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/meta", dir_fd:181, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924fdca38, flashback_version:0}, log_storage:{log_tail:{lsn:2171156}, readable_log_tail:{lsn:2171156}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64933612, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, palf_epoch:4, last_purge_throttling_ts:-1, this:0x2ad924fdaf70}, [Reconfirm]={palf_id:1003, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924fda9b0}) [2025-01-08 00:50:56.134757] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=312][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:50:56.134774] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:50:56.134784] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:50:56.134795] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] current log_task status(palf_id=1003, self="192.168.159.137:2882", log_id=17838, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:50:56.146070] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF41-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326256252335, keepalive_ts:-1}) [2025-01-08 00:50:56.146108] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF41-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326256252335, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:56.150162] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:56.161912] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC372F-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:56.165669] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC37A-0-0] [lt=154][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:56.168353] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:56.186845] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:50:56.197495] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C294-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:56.197543] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C294-0-0] [lt=31] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326255989509001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326255989509001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:56.203139] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=61] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:56.204093] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=20] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:56.205466] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:56.205565] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326254278994000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:56.212831] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=32] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:304K/70:177794 disk:304K/70:177794 t1m2:56K/12:446241 t1m2g0:56K/12:446241 t1002m2:176K/42:114842 t1002m2g0:176K/42:114842 t1001m2:72K/16:141708 t1001m2g0:72K/16:141708 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:56.216027] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=37] gc stale ls task succ [2025-01-08 00:50:56.222862] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:56.222915] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:56.225691] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D170BBD8D-0-0] [lt=90] get number(ret=0, service_type_=0, range=1, base_id=1736326256224732000, start_id=1736326256224732000, end_id=1736326256224732001) [2025-01-08 00:50:56.227793] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.229708] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5ADD-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=5430) [2025-01-08 00:50:56.232606] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8702E-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:56.232651] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:56.232663] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:56.233951] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58358][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:56.234156] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:50:56.247316] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=48] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8576, offset=2432862}, end_cursor:ObLogCursor{file_id=1, log_id=8577, offset=2433167}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946f0050, len:4258, log_data_len:305, seq:8576, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:56.248159] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E41-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:56.248283] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E41-0-0] [lt=103] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326256104493000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326256104493000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:56.252550] INFO [LIB] stat (utility.h:1210) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E41-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:56.263893] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17750, miss_cnt:0, cache_read_size:2171400, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:56.265522] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=75] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:56.267024] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=41] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:56.267121] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=25] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326255686329000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:56.268764] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8577, offset=2433167}, end_cursor:ObLogCursor{file_id=1, log_id=8578, offset=2433464}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946f4050, len:3953, log_data_len:297, seq:8577, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:56.269321] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E41-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:56.269364] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E41-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326256104493000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326256104493000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:56.280013] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] get number(ret=0, service_type_=0, range=1, base_id=1736326256279520000, start_id=1736326256279520000, end_id=1736326256279520001) [2025-01-08 00:50:56.280934] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=56] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:56.282178] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:50:56.288360] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=57] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:56.293369] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9E-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:56.293403] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9E-0-0] [lt=31] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:56.295747] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [46384][T1_L0_G0][T1][YB42C0A89F8B-00062B2D16DBAD83-0-0] [lt=21] sql stat record not found(ret=-5138, key={sql_id:"E268830A30994BFD0285516859916A6", plan_hash:3290318591324336132, source_addr:"192.168.159.139:45082"}) [2025-01-08 00:50:56.296341] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58356][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BB3-0-0] [lt=23] init point thread id with(&point=0x5572d50e2640, idx_=1012, point=[thread id=58356, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58356) [2025-01-08 00:50:56.297744] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8578, offset=2433464}, end_cursor:ObLogCursor{file_id=1, log_id=8579, offset=2433761}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946f8050, len:3656, log_data_len:297, seq:8578, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:56.299069] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8579, offset=2433761}, end_cursor:ObLogCursor{file_id=1, log_id=8580, offset=2434058}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946fc050, len:3359, log_data_len:297, seq:8579, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:56.305604] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185413-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:56.306387] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D16DBAD85-0-0] [lt=48] sql stat record not found(ret=-5138, key={sql_id:"E268830A30994BFD0285516859916A6", plan_hash:3290318591324336132, source_addr:"192.168.159.139:45058"}) [2025-01-08 00:50:56.319447] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.319494] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.327968] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.353320] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46384][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB5E-0-0] [lt=38][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193761267097, cache_obj->added_lc()=false, cache_obj->get_object_id()=13564, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:56.359427] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=37] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:56.360603] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=30] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:56.360696] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326255577817000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:56.360723] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326255577817000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326255577817000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:56.365393] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8FFF-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=2017, avg_log_submit_to_slide_time=23332) [2025-01-08 00:50:56.365521] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:56.367745] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:50:56.369907] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:56.370051] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=87, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:56.381261] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A446F-0-0] [lt=28][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17925, log_task={header:{begin_lsn:{lsn:2181725}, end_lsn:{lsn:2181847}, log_id:17925, min_scn:{val:1736326255989509002, v:0}, max_scn:{val:1736326255989509002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2181603}, prev_proposal_id:1, committed_end_lsn:{lsn:2181725}, data_checksum:1715074492, accum_checksum:4270300806, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326256280495, freeze_ts:1736326256280495, submit_ts:1736326256280508, flushed_ts:1736326256289982, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:9474}, fs_cb_begin_ts=1736326256381177, log_life_time=100682) [2025-01-08 00:50:56.383435] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83B9-0-0] [lt=28] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=2775, avg_log_submit_to_slide_time=46305) [2025-01-08 00:50:56.383643] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:56.387078] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=118] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:50:56.395757] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AC39-0-0] [lt=95] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.718940544128417969e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=385, Throttle Trigger(MB)=231, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.906445290688600380e+02) [2025-01-08 00:50:56.397335] INFO [STORAGE] runTimerTask (ob_ddl_heart_beat_task.cpp:49) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=19] send to rs all task status succeed [2025-01-08 00:50:56.398131] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=75] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2087, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:56.398244] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=110] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10336) [2025-01-08 00:50:56.398286] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=311, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:56.398389] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=87] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=31, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:56.399229] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:56.399260] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-01-08 00:50:56.399269] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:56.399282] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:56.399296] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193760312874, flush_io_finished_ret_:0, flush_io_finished_round_:3743, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:56.399808] INFO [LIB] stat (utility.h:1210) [58356][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA384D-0-0] [lt=32] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:56.399848] INFO [LIB] stat (utility.h:1210) [58356][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA384D-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4239, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:56.399859] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58356][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA384D-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=112, avg_log_gen_to_submit_time=208, avg_log_submit_to_flush_time=2644, avg_log_submit_to_slide_time=4031) [2025-01-08 00:50:56.400135] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=81] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10336, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:56.400412] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=51] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326255491315003, v:0}, new_max_version={val:1736326255491315003, v:0}, last_min_version={val:1736326255491315002, v:0}, last_max_version={val:1736326255491315002, v:0}, total_time_us=6581, delta_us=909095) [2025-01-08 00:50:56.400460] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326255491315003, v:0}, min_version={val:1736326255491315003, v:0}, max_version={val:1736326255491315003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:56.428116] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.432387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5990) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=67] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000) [2025-01-08 00:50:56.432422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.432552] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.432626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.432702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.432815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.432882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.432942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433003] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433064] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433124] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433184] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=100] get multi version start(ls_id={id:1}, tablet_id={id:101011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:101109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433499] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433688] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101098}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.433985] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434045] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434105] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434165] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434297] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:101036}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434359] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434492] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=77] get multi version start(ls_id={id:1}, tablet_id={id:60130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434554] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101064}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434614] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101080}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.434973] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435091] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101083}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435292] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:60263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435353] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435413] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435473] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101018}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435592] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435652] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=64] get multi version start(ls_id={id:1}, tablet_id={id:101034}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.435954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436075] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436185] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101084}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436301] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436504] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436624] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101068}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=72] get multi version start(ls_id={id:1}, tablet_id={id:515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436924] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.436984] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.437044] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.437171] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.437309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:60504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.437541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.437615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.437753] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.437822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.437885] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.437997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101073}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438121] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438244] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438314] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101099}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101092}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438552] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.438977] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439037] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101031}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=68] get multi version start(ls_id={id:1}, tablet_id={id:60464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439280] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:60425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439340] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439400] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=72] get multi version start(ls_id={id:1}, tablet_id={id:101111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439835] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.439954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440014] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440073] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101067}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440132] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440281] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440350] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101027}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440529] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440653] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440779] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.440953] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441073] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441425] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=169] get multi version start(ls_id={id:1}, tablet_id={id:60329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441486] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101082}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441547] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441634] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.441925] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.442035] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.442103] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.442162] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.444950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:60147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445102] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445165] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101057}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=93] get multi version start(ls_id={id:1}, tablet_id={id:101001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445501] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445561] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445621] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101033}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.445919] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446024] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101066}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446092] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446152] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446212] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446271] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446330] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446389] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446577] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446637] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101030}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101020}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446805] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446869] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101044}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:100002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.446989] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447049] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447167] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447313] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=89] get multi version start(ls_id={id:1}, tablet_id={id:293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447495] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447676] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101013}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447736] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447795] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447915] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.447975] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448034] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101072}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448094] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448153] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101088}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448283] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448348] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101069}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448408] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448468] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448566] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BEC3-0-0] [lt=40] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:50:56.448588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448648] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448767] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101060}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448826] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101100}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.448946] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449006] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101022}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449248] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449378] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449438] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449558] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449736] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449796] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449915] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.449975] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450035] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101035}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450094] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:5}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450438] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450507] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101058}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450644] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101081}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101086}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101052}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.450941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451001] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101019}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101079}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451243] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451312] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451491] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451550] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.451991] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.452098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.452166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.452296] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:60407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.452356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.452737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=100] get multi version start(ls_id={id:1}, tablet_id={id:50425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.452953] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453106] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=85] get multi version start(ls_id={id:1}, tablet_id={id:60148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453490] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=60] get multi version start(ls_id={id:1}, tablet_id={id:426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453625] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453691] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453753] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453812] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.453994] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.454057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.454118] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.454178] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.454351] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=96] get multi version start(ls_id={id:1}, tablet_id={id:50423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.454423] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.454489] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.455833] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=59] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=7879, avg_group_log_size=562) [2025-01-08 00:50:56.455870] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=7879) [2025-01-08 00:50:56.455889] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:56.456586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.456856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=199] get multi version start(ls_id={id:1}, tablet_id={id:60244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.456959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:60439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.457127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=106] get multi version start(ls_id={id:1}, tablet_id={id:60469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.457301] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=54] get multi version start(ls_id={id:1}, tablet_id={id:50419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.457401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:60003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.457494] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.457587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.457679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.457770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.457862] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.457966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:50157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458264] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=100] get multi version start(ls_id={id:1}, tablet_id={id:60149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458336] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458516] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458575] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458634] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458812] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.458931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101043}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459027] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459282] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:60121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459403] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101032}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.459940] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460000] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460179] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460306] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=71] get multi version start(ls_id={id:1}, tablet_id={id:50331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460358] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.460367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460486] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460547] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460606] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460786] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460845] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460905] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.460964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101028}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461024] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461269] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461336] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101021}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461547] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461612] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461732] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461910] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.461969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462354] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462594] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101037}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462784] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.462940] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101054}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463187] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=56] get multi version start(ls_id={id:1}, tablet_id={id:50005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463362] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101041}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463421] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101097}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463539] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101024}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463835] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.463955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464013] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464072] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464254] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101055}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464322] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464621] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464865] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.464986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101063}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.465216] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.465342] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.465538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:3}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.465734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=77] get multi version start(ls_id={id:1}, tablet_id={id:60112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.465800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.465861] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.465921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101040}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.465981] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466153] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466319] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466398] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466467] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101047}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466718] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466867] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:56.466906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.466967] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101059}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467075] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467140] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467341] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467402] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467462] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467521] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467639] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467699] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467877] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.467936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.468418] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.469229] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:56.469636] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.469802] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=103] get multi version start(ls_id={id:1}, tablet_id={id:380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.469865] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.469948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:60413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470259] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470388] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470448] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:60343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470868] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.470991] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471051] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101046}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471269] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471337] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471397] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471457] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471517] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471577] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471637] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471697] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471757] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471816] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471937] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.471997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101045}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472234] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472370] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472430] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472533] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472600] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472660] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472779] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101078}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.472957] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473135] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473239] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101049}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473370] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473430] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473604] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101101}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473663] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473783] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.473960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474079] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474284] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101091}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474354] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474474] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474533] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101039}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474652] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101029}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474830] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474889] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.474949] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475187] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475419] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=63] get multi version start(ls_id={id:1}, tablet_id={id:60261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475849] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.475970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476145] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:100004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476264] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476333] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101095}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101070}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476627] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476686] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476746] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101075}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476912] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.476978] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477038] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101077}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477323] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=100] get multi version start(ls_id={id:1}, tablet_id={id:328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477448] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101071}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477628] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477747] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101065}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477806] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.477985] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.478045] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.478104] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.478230] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.478295] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.478355] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.478416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.478476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101014}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.478535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.478595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.480760] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [50416][T1002_TimerWK1][T1001][YB42C0A89F89-00062B2D1D185391-0-0] [lt=24] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326256478462, can_tenant_elr:true}) [2025-01-08 00:50:56.482021] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=36] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:56.482530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=68] get multi version start(ls_id={id:1}, tablet_id={id:334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.482621] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:50443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.482689] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.482953] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2171522}], scn_array=[cnt:1, {val:1736326256208392002, v:0}], curr_size=122, accum_size=488, time_cost=14499) [2025-01-08 00:50:56.482752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.483755] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18649427}], scn_array=[cnt:1, {val:1736326256424079000, v:0}], curr_size=122, accum_size=1033, time_cost=13013) [2025-01-08 00:50:56.484437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=945] get multi version start(ls_id={id:1}, tablet_id={id:147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.484517] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:101102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.485167] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.485552] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=303] get multi version start(ls_id={id:1}, tablet_id={id:50505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.485789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=176] get multi version start(ls_id={id:1}, tablet_id={id:101004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.486059] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=24] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:56.487103] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=1155] get multi version start(ls_id={id:1}, tablet_id={id:60394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487182] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=130] get multi version start(ls_id={id:1}, tablet_id={id:60319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487559] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487680] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487740] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487861] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.487983] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488043] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488103] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488314] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101050}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488503] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:100001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488622] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488740] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488799] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488903] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.488971] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101056}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489150] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489270] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489334] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101023}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489600] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489705] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:101048}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489766] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489826] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101026}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489885] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.489945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490064] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490123] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490182] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490494] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490554] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490614] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101025}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490733] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490852] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490911] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.490970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.491030] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:4}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.491089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.491148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.491309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.491393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.491462] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.491527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.491845] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.491998] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492123] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:50398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492187] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492326] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=77] get multi version start(ls_id={id:1}, tablet_id={id:50259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492391] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492453] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492515] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492577] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101085}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492699] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492759] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.492942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493002] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101087}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493254] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493478] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493662] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101051}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.493968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101061}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494286] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494353] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494414] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101076}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494656] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=61] get multi version start(ls_id={id:1}, tablet_id={id:50168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494843] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494903] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.494965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495251] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495319] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495381] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101074}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495802] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101053}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495925] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.495985] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.496046] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.496106] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.496166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.497543] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84201813}], scn_array=[cnt:1, {val:1736326256478462000, v:0}], curr_size=586, accum_size=5723, time_cost=15352) [2025-01-08 00:50:56.496305] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=80] get multi version start(ls_id={id:1}, tablet_id={id:60142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.497870] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=1490] get multi version start(ls_id={id:1}, tablet_id={id:50130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.497969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.498062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.498153] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.498317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.498415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.498562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.498666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:60474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.498758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.498915] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499101] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499191] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499359] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=110] get multi version start(ls_id={id:1}, tablet_id={id:505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499543] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499641] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:60295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.499998] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.500088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.500283] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.500405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:60169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.500523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=56] get multi version start(ls_id={id:1}, tablet_id={id:101042}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.500710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=52] get multi version start(ls_id={id:1}, tablet_id={id:145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.500814] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:50328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.500911] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.501062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.501160] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.501376] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.501484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:50293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.501581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101096}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.501677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.501781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:60247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.501875] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.501968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502314] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:60222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502795] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502888] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:101038}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.502980] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.503072] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.503337] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=193] get multi version start(ls_id={id:1}, tablet_id={id:60260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.503441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:50169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.503540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.503644] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:101090}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.503739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.503845] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.503952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.504045] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324455577817000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324445564663000, max_merged_snapshot=1) [2025-01-08 00:50:56.504324] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:116) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541F-0-0] [lt=39] SSTableGCTask(cost_ts=72311) [2025-01-08 00:50:56.505377] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2181969}], scn_array=[cnt:1, {val:1736326255989509004, v:0}], curr_size=122, accum_size=732, time_cost=22542) [2025-01-08 00:50:56.519070] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.528329] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.528652] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=85] PNIO [ratelimit] time: 1736326256528648, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006973, add_bytes: 0 [2025-01-08 00:50:56.542154] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.544056] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.565792] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58135][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB189-0-0] [lt=15] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:50:56.565865] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58135][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB189-0-0] [lt=63] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:56.565900] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58135][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB189-0-0] [lt=33] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:50:56.570553] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=59] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:56.570872] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=66, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:56.570915] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:56.570936] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:56.570951] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:56.572992] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=51] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:56.573330] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EA-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:56.573373] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EA-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326256208392003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326256208392003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:56.574492] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83BD-0-0] [lt=64] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:56.575005] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:56.575031] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5025, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:56.575737] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9006-0-0] [lt=56] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:56.575758] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9006-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5329, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:56.575767] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9006-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=111, avg_log_submit_to_flush_time=3580, avg_log_submit_to_slide_time=5217) [2025-01-08 00:50:56.576026] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=52] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:56.579271] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD0-0-0] [lt=25] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:56.579302] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD0-0-0] [lt=25] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=33) [2025-01-08 00:50:56.588284] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:50:56.588502] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:56.589942] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58360][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:56.590316] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1002, group_id_=1, token=5) [2025-01-08 00:50:56.602945] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info(lsn={lsn:2182213}, scn={val:1736326256491846001, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326256589850, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2182213}, next_to_submit_scn_:{val:1736326256491846001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2182091}, end_lsn:{lsn:2182213}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2182213}, readable_log_tail:{lsn:2182213}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64922555, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326256491846000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326256491846000, v:0}, data_checksum:3348837161, flag:46881}}, init_mode_version:0, accumulate_checksum:2878628097, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326256491846000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17840, miss_cnt:0, cache_read_size:2182213, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:56.604786] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=23] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:56.628393] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.645093] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B91-0-0] [lt=31] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:56.652436] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=78] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:56.660352] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5AE3-0-0] [lt=37] init point thread id with(&point=0x5572d50e26c0, idx_=1014, point=[thread id=58358, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58358) [2025-01-08 00:50:56.670821] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=51] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5189895787, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:56.670884] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:56.670900] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8df664d70) [2025-01-08 00:50:56.670910] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5189895787, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:56.672701] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2947, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:56.672738] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1141, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:56.676940] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9008-0-0] [lt=110] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:56.676966] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9008-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14370, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:56.682799] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4475-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:56.689320] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D685-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:56.689347] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D685-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=30141, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:56.695835] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=535] get_max_applied_scn(scn={val:1736326256424079001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18649793}, palf_committed_end_scn_:{val:1736326256424079002, v:0}, last_check_scn_:{val:1736326256424079002, v:0}, max_applied_cb_scn_:{val:1736326256424079001, v:0}}) [2025-01-08 00:50:56.695880] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=41] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18649793}, readable_log_tail:{lsn:18649793}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48454975, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:56.696046] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=83] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:56.705740] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [50457][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=22] XA scheduler heartbeat task statistics(avg_time=1) [2025-01-08 00:50:56.710511] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8A-00062B2D1B49C72D-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326256709695000, start_id=1736326256709695000, end_id=1736326256709695001) [2025-01-08 00:50:56.728534] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=111][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.761299] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC373B-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:56.761366] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC373B-0-0] [lt=34] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:56.767304] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:56.771086] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:56.779307] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=35] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:56.790004] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:50:56.790633] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=15] ====== tenant freeze timer task ====== [2025-01-08 00:50:56.791973] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50625][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D014-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:56.795962] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326255989509004, v:0}, base_version={val:1736326255989509002, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326256208392004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326256795933, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326255989509004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326256888510, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326255989509004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326256609095, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:50:56.798001] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=10410, avg_group_log_size=650) [2025-01-08 00:50:56.798023] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10410) [2025-01-08 00:50:56.812165] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=122] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:56.828166] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=51] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:56.828217] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=90] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:56.828251] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=25] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:56.828256] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=77] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=176) [2025-01-08 00:50:56.828314] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=60] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=172) [2025-01-08 00:50:56.828669] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=118][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.828893] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [50576][T1001_RFLWorker][T1001][YB42C0A89F89-00062B2D1C78458E-0-0] [lt=37] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:50:56.835048] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A70-0-0] [lt=40] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326256835015}) [2025-01-08 00:50:56.835172] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285950-0-0] [lt=30] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:56.842110] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285950-0-0] [lt=82] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326256842086, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326256842086, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326256842086, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:56.842157] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285950-0-0] [lt=46] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326256835015}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326256842086, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326256842086, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326256842086, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:56.845717] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=33] LogLoopThread round_cost_time(us)(round_cost_time=12) [2025-01-08 00:50:56.851087] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=27] start to print blacklist info [2025-01-08 00:50:56.851804] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=349] ls blacklist refresh finish(cost_time=18146) [2025-01-08 00:50:56.857871] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=2861, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:56.857901] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=7098) [2025-01-08 00:50:56.857954] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=316, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:56.857991] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=28, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:56.858086] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=49] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1003}, {id:1001}]) [2025-01-08 00:50:56.859548] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5AE6-0-0] [lt=28] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=29, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:56.859587] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5AE6-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=4844, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:56.859608] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5AE6-0-0] [lt=17] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=3228, avg_log_submit_to_slide_time=4734) [2025-01-08 00:50:56.859760] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=7098, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:56.871314] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1736326256871309, bytes: 868011636, bw: 0.281632 MB/s, add_ts: 1003989, add_bytes: 296491 [2025-01-08 00:50:56.876232] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2188, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:56.884081] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4479-0-0] [lt=19] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:56.884134] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4479-0-0] [lt=51] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:56.886298] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4715, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:56.886348] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:56.886371] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=191, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:56.886519] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=94, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:56.886968] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=27] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:50:56.895448] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0de8db0, session_id:3221577800, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326238256851, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1496, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:56.895508] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=49] kill and revert session(conn.sessid_=3221577800, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:50:56.895522] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sock ref clean, do destroy(*s={this:0x2ad8a0de8db0, session_id:3221577800, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326238256851, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1496, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:56.895561] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221577800, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:50:56.895574] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221577800, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B70-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:18, cur_pkt_pos_:24, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:3623465, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:3623466, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:3623467, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:3623467, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:3623467, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:3623467, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:3623467, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:13}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:3623467, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:14}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:3623467, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:15}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{receive:obp20_header_:{payload_len:194, pkt_seq:0, req_id:3623468, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:190, rec_:194, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[11]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:3623468, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:3623468, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[13]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:3623468, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[14]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:3623468, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[15]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:3623468, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[16]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:3623468, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:13}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[17]:{receive:obp20_header_:{payload_len:410, pkt_seq:0, req_id:3623469, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:406, rec_:410, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[18]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:3623469, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[19]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:3623469, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[20]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:3623469, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[21]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:3623469, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:1, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[22]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:3623469, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[23]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:3623469, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:10}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:56.895964] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=99] free session successfully(sessid=3221577800) [2025-01-08 00:50:56.896080] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=114] mark session id unused(sessid=3221577800) [2025-01-08 00:50:56.896072] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=405] ====== check clog disk timer task ====== [2025-01-08 00:50:56.896101] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:50:56.897269] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.897268] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.897323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.897349] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:56.904081] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=29] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:50:56.904124] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=40] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=6) [2025-01-08 00:50:56.904146] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=21] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:50:56.904157] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=7] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:50:56.924575] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9821) [2025-01-08 00:50:56.925728] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=203] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326256925708], gts:1736326256925355000, latest_srr:[mts=1736326256925708]}) [2025-01-08 00:50:56.925759] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:56.925782] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326256925771], gts:1736326256925355000, latest_srr:[mts=1736326256925771]}) [2025-01-08 00:50:56.926064] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326256926050], gts:1736326256925355000, latest_srr:[mts=1736326256926050]}) [2025-01-08 00:50:56.928798] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:56.929863] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C19C277-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1736326256927923000, start_id=1736326256927923000, end_id=1736326256927923001) [2025-01-08 00:50:56.929905] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] ====== check clog disk timer task ====== [2025-01-08 00:50:56.929957] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=46] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:56.953673] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C88-0-0] [lt=63][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:56.956188] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C88-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:56.967271] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:56.967855] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856D0-0-0] [lt=44] InfoPoolResizeTask(cost_ts=8) [2025-01-08 00:50:56.969405] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:56.971601] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:56.971932] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:56.973391] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=89] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2066) [2025-01-08 00:50:56.973553] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:50:56.974038] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:56.981408] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:56.986350] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326246976254, end_time=1736326256986347, dag_status={}) [2025-01-08 00:50:56.986384] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=29] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:56.986398] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:56.986406] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:56.986673] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=264] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:56.986684] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:56.986693] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:56.986700] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:56.986859] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=158] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:56.986920] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:56.986928] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.986940] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.986949] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987009] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987019] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987027] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987035] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987043] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987051] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987059] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987067] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987075] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987083] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987091] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987099] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987107] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987115] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987122] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987131] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987139] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987146] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987155] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987162] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987170] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987178] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987186] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987257] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987268] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987276] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987284] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987292] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987300] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987308] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987316] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987331] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987339] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987347] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987355] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987363] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987431] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987440] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987449] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987456] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987464] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987472] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987480] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987487] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987539] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=50] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987548] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987656] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=106] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987671] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987679] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987687] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987694] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987702] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987710] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987718] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987725] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987733] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987741] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987749] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987757] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987764] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987798] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=33] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987807] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987815] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987823] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987831] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987838] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987846] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987854] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987862] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:56.987872] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:50:56.987880] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:50:56.987887] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:50:56.987892] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:50:56.987897] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:50:56.987903] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:50:56.987908] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:50:56.987913] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:50:56.987918] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:50:56.987923] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:50:56.987929] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:50:56.988061] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17844, miss_cnt:0, cache_read_size:2182701, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:56.988392] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=81] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=178, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:56.988519] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=125] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:56.990333] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=73] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:50:57.017894] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D31-0-0] [lt=15] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:57.022413] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=8] Garbage Collector is running(seq_=193, gc_interval=10000000) [2025-01-08 00:50:57.022476] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=38] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:50:57.022493] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=14] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:50:57.023791] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:50:57.025303] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=37] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:50:57.025329] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=21] gc_check_member_list_ cost time(ret=0, time_us=2878) [2025-01-08 00:50:57.025345] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:50:57.027186] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:57.029167] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=84] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:57.029504] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.030653] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=87] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:57.031952] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:57.031977] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:57.031993] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=15] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:57.032005] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:57.032016] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:57.032027] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=50) [2025-01-08 00:50:57.032035] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:50:57.032043] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-01-08 00:50:57.032062] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-01-08 00:50:57.042364] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB60-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193761956193, cache_obj->added_lc()=false, cache_obj->get_object_id()=13565, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.042639] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:57.042760] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4726, iops:13, bw:60873, [delay/us]:prepare:12, schedule:245, submit:450, rt:1990, total:2699) [2025-01-08 00:50:57.042788] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:57.042846] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4726, avg_iops:13, avg_bw:60871, [delay/us]: prepare:12, schedule:245, submit:450, device:1990, total:2699) [2025-01-08 00:50:57.042861] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:57.042901] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:73045, [delay/us]:prepare:8, schedule:149, submit:269, rt:1436, total:1864) [2025-01-08 00:50:57.042912] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:57.042948] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:73050, [delay/us]: prepare:8, schedule:149, submit:269, device:1436, total:1864) [2025-01-08 00:50:57.042959] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:57.043035] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20291, [delay/us]:prepare:5, schedule:1295, submit:1, rt:1091, total:2394) [2025-01-08 00:50:57.043062] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4198, iops:40, bw:166386, [delay/us]:prepare:16, schedule:488, submit:889, rt:1622, total:3016) [2025-01-08 00:50:57.043072] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:57.043105] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20292, [delay/us]: prepare:5, schedule:1295, submit:1, device:1091, total:2394) [2025-01-08 00:50:57.043121] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4198, avg_iops:40, avg_bw:166400, [delay/us]: prepare:16, schedule:488, submit:889, device:1622, total:3016) [2025-01-08 00:50:57.043225] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=97] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:57.043249] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4058, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:7830, total:7843) [2025-01-08 00:50:57.043275] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:57.043307] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4058, [delay/us]: prepare:0, schedule:0, submit:600000000, device:7830, total:7843) [2025-01-08 00:50:57.043325] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:57.043341] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:57.043350] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:57.043358] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:57.043366] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=294kB/s, net_out=268kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:57.056837] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=74, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:57.062310] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3741-0-0] [lt=14][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17846, log_task={header:{begin_lsn:{lsn:2172132}, end_lsn:{lsn:2172254}, log_id:17846, min_scn:{val:1736326256709695003, v:0}, max_scn:{val:1736326256709695003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2172010}, prev_proposal_id:1, committed_end_lsn:{lsn:2172132}, data_checksum:940009244, accum_checksum:1592078635, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326256961928, freeze_ts:1736326256961928, submit_ts:1736326256961941, flushed_ts:1736326256963412, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:1471}, fs_cb_begin_ts=1736326257062295, log_life_time=100367) [2025-01-08 00:50:57.063825] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16D9BA1B-0-0] [lt=20] sql stat record not found(ret=-5138, key={sql_id:"E268830A30994BFD0285516859916A6", plan_hash:3290318591324336132, source_addr:"192.168.159.138:42022"}) [2025-01-08 00:50:57.067031] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46384][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16D9BA1C-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1736326257066909000, start_id=1736326257066909000, end_id=1736326257066909001) [2025-01-08 00:50:57.068363] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=42] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:57.068400] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=85) [2025-01-08 00:50:57.071780] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=111] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=2066, avg_group_log_size=206) [2025-01-08 00:50:57.071816] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2066) [2025-01-08 00:50:57.072155] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=63] new mysql sessid created(conn.sessid_=3221580517, support_ssl=false) [2025-01-08 00:50:57.072367] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221580517, sess.client_addr_="192.168.159.136:39638") [2025-01-08 00:50:57.072404] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x2ad8a0dfc030, session_id:3221580517, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:156}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.072647] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=78] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:57.073095] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=46] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.073112] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:292) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=18] mysql login queue(mysql_queue->queue_.size()=1) [2025-01-08 00:50:57.073234] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A6F-0-0] [lt=24] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221580517, conn.client_addr_port_=39638, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221580517, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221580517, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D17784A6F-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46334, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523214408521997, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:74, fixup_index:-1, fixup_ash_buffer:{this:0x2ad8c1dd2af8, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:50:57.075383] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A6F-0-0] [lt=91][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:50:57.075452] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A6F-0-0] [lt=18] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221580517, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.075593] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:57.075714] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=110] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3784, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:57.075770] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2172254}], scn_array=[cnt:1, {val:1736326256709695004, v:0}], curr_size=122, accum_size=1220, time_cost=13169) [2025-01-08 00:50:57.075790] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:57.075807] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=802, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:57.076306] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:57.081168] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=23] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:57.081547] INFO [SERVER] process (obmp_init_db.cpp:53) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84A9F-0-0] [lt=7] init db(db_name=oceanbase) [2025-01-08 00:50:57.082669] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=19] rpc statistics(total_trans_msg_count=3, total_batch_msg_count=0) [2025-01-08 00:50:57.083599] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2149675}], scn_array=[cnt:1, {val:1736326256925355001, v:0}], curr_size=122, accum_size=122, time_cost=10981) [2025-01-08 00:50:57.083689] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:57.083724] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:57.085917] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:57.089162] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=82] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:57.091686] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CC1-0-0] [lt=44] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10006, add_timestamp:1736326257091669}) [2025-01-08 00:50:57.092343] WDIAG [SHARE] inner_to_show_str (ob_system_variable.cpp:516) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CC1-0-0] [lt=27][errcode=0] sys var casted obj is not ObVarcharType(ret=0, value={"NULL":"NULL"}, *res_obj={"NULL":"NULL"}, get_name()=secure_file_priv) [2025-01-08 00:50:57.092550] WDIAG [SHARE] inner_to_show_str (ob_system_variable.cpp:516) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CC1-0-0] [lt=22][errcode=0] sys var casted obj is not ObVarcharType(ret=0, value={"NULL":"NULL"}, *res_obj={"NULL":"NULL"}, get_name()=external_user) [2025-01-08 00:50:57.093692] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285951-0-0] [lt=26] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10006, locations=[{table_id:10006, partition_id:10006, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257093669, sql_renew_time:1736326257093669, is_mark_fail:false}]) [2025-01-08 00:50:57.093729] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285951-0-0] [lt=35] success to process renew task(task={tenant_id:1, table_id:10006, add_timestamp:1736326257091669}, locations=[{table_id:10006, partition_id:10006, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257093669, sql_renew_time:1736326257093669, is_mark_fail:false}]) [2025-01-08 00:50:57.095954] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16D9BA1D-0-0] [lt=324] sql stat record not found(ret=-5138, key={sql_id:"E268830A30994BFD0285516859916A6", plan_hash:3290318591324336132, source_addr:"192.168.159.138:42000"}) [2025-01-08 00:50:57.098639] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2186797, this={dio_aligned_buf:{buf_write_offset:3751, buf_padding_size:345, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:26195, truncate_used_ts:5386}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2183168, buf_len=122, total_write_size=12050, total_write_size_after_dio=417792, ob_pwrite_used_ts=252282, count=99, each_pwrite_cost=2548) [2025-01-08 00:50:57.099512] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221580520, support_ssl=false) [2025-01-08 00:50:57.099715] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=91] sm conn init succ(conn.sessid_=3221580520, sess.client_addr_="192.168.159.136:39642") [2025-01-08 00:50:57.099737] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x2ad8efff8030, session_id:3221580520, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:183}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.099801] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=30] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84206140}, readable_log_tail:{lsn:84206140}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:50003396, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:57.099889] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=86] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:57.099921] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:50:57.100503] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.100569] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A74-0-0] [lt=95] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221580520, conn.client_addr_port_=39642, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221580520, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221580520, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D17584A74-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46333, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523214482586774, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:61, fixup_index:-1, fixup_ash_buffer:{this:0x2ad8cdd88af8, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:50:57.102473] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A74-0-0] [lt=60][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:50:57.102542] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A74-0-0] [lt=18] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221580520, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.104373] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46390][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185785-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193762018205, cache_obj->added_lc()=false, cache_obj->get_object_id()=13568, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.109462] INFO [SERVER] process (obmp_init_db.cpp:53) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CC2-0-0] [lt=28] init db(db_name=oceanbase) [2025-01-08 00:50:57.109729] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185786-0-0] [lt=49][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:57.113243] WDIAG get_refactored (ob_hashtable.h:1120) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185788-0-0] [lt=16][errcode=-4201] hashtable not init [2025-01-08 00:50:57.115017] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CC5-0-0] [lt=16] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12358, add_timestamp:1736326257115002}) [2025-01-08 00:50:57.115078] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CC5-0-0] [lt=41] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12396, add_timestamp:1736326257115076}) [2025-01-08 00:50:57.118741] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CC5-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.119038] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285952-0-0] [lt=17] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12358, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257119016, sql_renew_time:1736326257119016, is_mark_fail:false}]) [2025-01-08 00:50:57.119131] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285952-0-0] [lt=91] success to process renew task(task={tenant_id:1, table_id:12358, add_timestamp:1736326257115002}, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257119016, sql_renew_time:1736326257119016, is_mark_fail:false}]) [2025-01-08 00:50:57.119165] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285953-0-0] [lt=16] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12396, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257119157, sql_renew_time:1736326257119157, is_mark_fail:false}]) [2025-01-08 00:50:57.119285] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285953-0-0] [lt=118] success to process renew task(task={tenant_id:1, table_id:12396, add_timestamp:1736326257115076}, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257119157, sql_renew_time:1736326257119157, is_mark_fail:false}]) [2025-01-08 00:50:57.119447] INFO [STORAGE.BLKMGR] mark_macro_blocks (ob_block_manager.cpp:1076) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=47] no block alloc/free, no need to mark blocks(ret=0, mark_info.count()=0) [2025-01-08 00:50:57.119499] INFO [STORAGE.BLKMGR] mark_and_sweep (ob_block_manager.cpp:1056) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=49] finish once mark and sweep(ret=0, alloc_num=2962, mark_info.count()=0, marker_status={total_block_count:1024, reserved_block_count:2, linked_block_count:0, tmp_file_count:0, data_block_count:11, shared_data_block_count:11, index_block_count:0, ids_block_count:0, disk_block_count:0, bloomfiter_count:0, hold_count:41, pending_free_count:1, free_count:981, shared_meta_block_count:19, mark_cost_time:163, sweep_cost_time:0, start_time:"2025-01-08 00:50:57.119333", last_end_time:"2025-01-08 00:20:57.6873", hold_info:nothing, mark_finished:false}, map_cnt=41) [2025-01-08 00:50:57.121015] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CC8-0-0] [lt=25] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11001, add_timestamp:1736326257120989}) [2025-01-08 00:50:57.121382] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285954-0-0] [lt=18] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11001, vtable_type={tenant_id:1, type:2}) [2025-01-08 00:50:57.122282] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285954-0-0] [lt=24] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11001, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257122179, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.122318] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285954-0-0] [lt=36] success to process renew task(task={tenant_id:1, table_id:11001, add_timestamp:1736326257120989}, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257122179, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.127934] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CC8-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.129636] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.130711] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=85, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:57.137919] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:57.140299] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58241][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:50:57.142662] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326253112995, last_first_register_time_us=1736324454612308, register_time_us=1736326253112995, curr_time_us=1736326257142622) [2025-01-08 00:50:57.142835] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326257142695, last_submit_register_req_time_us=1736326257142695, register_parent_reason=1) [2025-01-08 00:50:57.142858] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:50:57.143006] WDIAG [SQL.RESV] get_table_schema (ob_schema_checker.cpp:889) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA2-0-0] [lt=29][errcode=-4002] invalid arguments(table_id=18446744073709551615, ret=-4002, lbt()="0x810107f 0x13d10e0d 0x808ce2e 0x14effa7a 0x14fa61b9 0x828d01d 0x81c080b 0x15140419 0x82376fc 0x8235716 0x822a9dd 0x8380932 0x836e7b0 0x1510d393 0x13c52c58 0x7cdaf74 0x7a7b458 0x7a7a129 0x7a767e0 0x78e6cb7 0x78c122b 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.143051] WDIAG [SQL.RESV] get_table_schema (ob_schema_checker.cpp:889) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA2-0-0] [lt=19][errcode=-4002] invalid arguments(table_id=18446744073709551615, ret=-4002, lbt()="0x810107f 0x13d10e0d 0x808ce2e 0x14effa7a 0x14fa61b9 0x828d01d 0x81c080b 0x15140419 0x82376fc 0x8235716 0x822a9dd 0x8380932 0x836e7b0 0x1510d393 0x13c52c58 0x7cdaf74 0x7a7b458 0x7a7a129 0x7a767e0 0x78e6cb7 0x78c122b 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.146416] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=36] new mysql sessid created(conn.sessid_=3221580529, support_ssl=false) [2025-01-08 00:50:57.146473] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221580529, sess.client_addr_="127.0.0.1:51592") [2025-01-08 00:50:57.146491] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2ad8efff8db0, session_id:3221580529, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:193}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.146524] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221580530, support_ssl=false) [2025-01-08 00:50:57.146554] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221580530, sess.client_addr_="127.0.0.1:51596") [2025-01-08 00:50:57.146566] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efffadb0, session_id:3221580530, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:194}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.146590] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221580531, support_ssl=false) [2025-01-08 00:50:57.146618] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221580531, sess.client_addr_="127.0.0.1:51600") [2025-01-08 00:50:57.146629] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8efffc030, session_id:3221580531, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:195}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.146651] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221580532, support_ssl=false) [2025-01-08 00:50:57.146680] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221580532, sess.client_addr_="127.0.0.1:51610") [2025-01-08 00:50:57.146691] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efffcdb0, session_id:3221580532, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:196}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.146975] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221580533, support_ssl=false) [2025-01-08 00:50:57.147021] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221580533, sess.client_addr_="127.0.0.1:51590") [2025-01-08 00:50:57.147035] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8efe48030, session_id:3221580533, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:197}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.147059] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221580534, support_ssl=false) [2025-01-08 00:50:57.147088] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221580534, sess.client_addr_="127.0.0.1:51594") [2025-01-08 00:50:57.147100] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8efe48db0, session_id:3221580534, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:198}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.147121] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221580535, support_ssl=false) [2025-01-08 00:50:57.147150] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221580535, sess.client_addr_="127.0.0.1:51598") [2025-01-08 00:50:57.147161] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efe4a030, session_id:3221580535, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:200}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.147182] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221580536, support_ssl=false) [2025-01-08 00:50:57.147568] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=351] sm conn init succ(conn.sessid_=3221580536, sess.client_addr_="127.0.0.1:51604") [2025-01-08 00:50:57.147584] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x2ad8efe4adb0, session_id:3221580536, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:201}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.147617] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221580537, support_ssl=false) [2025-01-08 00:50:57.147650] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221580537, sess.client_addr_="127.0.0.1:51612") [2025-01-08 00:50:57.147663] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8efe4c030, session_id:3221580537, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:202}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.150458] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46380][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCA-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12276, add_timestamp:1736326257150429}) [2025-01-08 00:50:57.151815] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=183] new mysql sessid created(conn.sessid_=3221580538, support_ssl=false) [2025-01-08 00:50:57.151822] WDIAG [SQL.RESV] get_table_schema (ob_schema_checker.cpp:889) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185789-0-0] [lt=46][errcode=-4002] invalid arguments(table_id=18446744073709551615, ret=-4002, lbt()="0x810107f 0x13d10e0d 0x808ce2e 0x14effa7a 0x14fa61b9 0x828d01d 0x81c080b 0x15140419 0x82376fc 0x8235716 0x822a9dd 0x8380932 0x836e7b0 0x1510d393 0x13c52c58 0x7cdaf74 0x7a7b458 0x7a7a129 0x7a767e0 0x78e6cb7 0x78c122b 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.151868] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221580538, sess.client_addr_="127.0.0.1:51614") [2025-01-08 00:50:57.151872] WDIAG [SQL.RESV] get_table_schema (ob_schema_checker.cpp:889) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185789-0-0] [lt=22][errcode=-4002] invalid arguments(table_id=18446744073709551615, ret=-4002, lbt()="0x810107f 0x13d10e0d 0x808ce2e 0x14effa7a 0x14fa61b9 0x828d01d 0x81c080b 0x15140419 0x82376fc 0x8235716 0x822a9dd 0x8380932 0x836e7b0 0x1510d393 0x13c52c58 0x7cdaf74 0x7a7b458 0x7a7a129 0x7a767e0 0x78e6cb7 0x78c122b 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.151884] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x2ad8a0de8030, session_id:3221580538, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:203}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.151913] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221580539, support_ssl=false) [2025-01-08 00:50:57.151945] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221580539, sess.client_addr_="127.0.0.1:51620") [2025-01-08 00:50:57.151957] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8a0dea030, session_id:3221580539, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:204}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.151979] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221580540, support_ssl=false) [2025-01-08 00:50:57.152008] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221580540, sess.client_addr_="127.0.0.1:51622") [2025-01-08 00:50:57.152019] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8a0deadb0, session_id:3221580540, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:205}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.153202] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221580541, support_ssl=false) [2025-01-08 00:50:57.153311] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=74] sm conn init succ(conn.sessid_=3221580541, sess.client_addr_="127.0.0.1:51602") [2025-01-08 00:50:57.153326] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8a0de8db0, session_id:3221580541, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.153354] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] new mysql sessid created(conn.sessid_=3221580542, support_ssl=false) [2025-01-08 00:50:57.153385] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221580542, sess.client_addr_="127.0.0.1:51606") [2025-01-08 00:50:57.153397] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0dec030, session_id:3221580542, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:206}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.153419] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221580543, support_ssl=false) [2025-01-08 00:50:57.153448] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221580543, sess.client_addr_="127.0.0.1:51608") [2025-01-08 00:50:57.153459] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8a0decdb0, session_id:3221580543, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:207}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.153481] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221580544, support_ssl=false) [2025-01-08 00:50:57.153510] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221580544, sess.client_addr_="127.0.0.1:51616") [2025-01-08 00:50:57.153521] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0dee030, session_id:3221580544, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:208}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.153543] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221580545, support_ssl=false) [2025-01-08 00:50:57.153571] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221580545, sess.client_addr_="127.0.0.1:51618") [2025-01-08 00:50:57.153583] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8a0deedb0, session_id:3221580545, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:209}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.153605] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221580546, support_ssl=false) [2025-01-08 00:50:57.153634] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221580546, sess.client_addr_="127.0.0.1:51624") [2025-01-08 00:50:57.153645] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8a0df0030, session_id:3221580546, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:210}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.158327] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285955-0-0] [lt=15] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12276, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:57.158606] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58346][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF42-0-0] [lt=16] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326257254332, keepalive_ts:-1}) [2025-01-08 00:50:57.158629] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58346][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF42-0-0] [lt=21] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326257254332, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:57.159420] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:57.161997] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.162043] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.162063] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.162086] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.162102] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.162121] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.162136] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.162155] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.162327] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5B-0-0] [lt=22] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.163118] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5B-0-0] [lt=18] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580532, proxy_sessid=0, client_sessid=3221580532, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.163302] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5C-0-0] [lt=27] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.164420] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5C-0-0] [lt=16] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580531, proxy_sessid=0, client_sessid=3221580531, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.164510] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5D-0-0] [lt=32] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.164391] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50899][T1002_L0_G2][T1002][YB42C0A89F8B-00062B2D1EABD79E-0-0] [lt=95] get_min_unreplayed_log_info(lsn={lsn:2172254}, scn={val:1736326256709695004, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326257062301, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2172254}, next_to_submit_scn_:{val:1736326256709695004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2172132}, end_lsn:{lsn:2172254}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2172376}, readable_log_tail:{lsn:2172376}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64932392, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326256709695003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326256709695003, v:0}, data_checksum:2124447694, flag:51216}}, init_mode_version:0, accumulate_checksum:1592078635, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326256709695003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17757, miss_cnt:0, cache_read_size:2172254, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:57.165333] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5D-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580530, proxy_sessid=0, client_sessid=3221580530, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.165411] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5E-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.166054] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5E-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580529, proxy_sessid=0, client_sessid=3221580529, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.166941] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3743-0-0] [lt=64] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:57.166975] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3743-0-0] [lt=17][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17847, log_task={header:{begin_lsn:{lsn:2172254}, end_lsn:{lsn:2172376}, log_id:17847, min_scn:{val:1736326256709695004, v:0}, max_scn:{val:1736326256709695004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2172132}, prev_proposal_id:1, committed_end_lsn:{lsn:2172254}, data_checksum:940009244, accum_checksum:282062038, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326257062265, freeze_ts:1736326257062265, submit_ts:1736326257062279, flushed_ts:1736326257076268, gen_to_freeze cost time:0, gen_to_submit cost time:14, submit_to_flush cost time:13989}, fs_cb_begin_ts=1736326257166965, log_life_time=104700) [2025-01-08 00:50:57.167958] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=77] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.168177] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.168270] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=75] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.168440] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=147] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.168462] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.168483] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.168566] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=67] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.168653] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.168676] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.168697] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.168918] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A75-0-0] [lt=26] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.169049] WDIAG [SQL.RESV] get_table_schema (ob_schema_checker.cpp:889) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18578A-0-0] [lt=17][errcode=-4002] invalid arguments(table_id=18446744073709551615, ret=-4002, lbt()="0x810107f 0x13d10e0d 0x808ce2e 0x14effa7a 0x14fa61b9 0x828d01d 0x81c080b 0x15140419 0x82376fc 0x8235716 0x822a9dd 0x8380932 0x836e7b0 0x1510d393 0x13c52c58 0x7cdaf74 0x7a7b458 0x7a7a129 0x7a767e0 0x78e6cb7 0x78c122b 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.169097] WDIAG [SQL.RESV] get_table_schema (ob_schema_checker.cpp:889) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18578A-0-0] [lt=21][errcode=-4002] invalid arguments(table_id=18446744073709551615, ret=-4002, lbt()="0x810107f 0x13d10e0d 0x808ce2e 0x14effa7a 0x14fa61b9 0x828d01d 0x81c080b 0x15140419 0x82376fc 0x8235716 0x822a9dd 0x8380932 0x836e7b0 0x1510d393 0x13c52c58 0x7cdaf74 0x7a7b458 0x7a7a129 0x7a767e0 0x78e6cb7 0x78c122b 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.170406] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A75-0-0] [lt=19] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580537, proxy_sessid=0, client_sessid=3221580537, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.170496] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A76-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.171306] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A76-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580536, proxy_sessid=0, client_sessid=3221580536, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.171452] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A77-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.172594] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A77-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580535, proxy_sessid=0, client_sessid=3221580535, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.172679] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A78-0-0] [lt=24] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.173464] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A78-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580534, proxy_sessid=0, client_sessid=3221580534, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.173544] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A79-0-0] [lt=24] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.174561] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A79-0-0] [lt=11] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580533, proxy_sessid=0, client_sessid=3221580533, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.176301] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.176366] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=38] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.176388] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.176448] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=28] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.176470] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.176490] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.176505] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.176525] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.176540] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.176557] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.176571] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.176587] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.176617] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.176642] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.176657] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.176673] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.176663] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A70-0-0] [lt=27] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.176688] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:50:57.176707] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:50:57.177602] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A70-0-0] [lt=19] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580546, proxy_sessid=0, client_sessid=3221580546, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.177762] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A71-0-0] [lt=24] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.178730] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A71-0-0] [lt=14] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580545, proxy_sessid=0, client_sessid=3221580545, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.178913] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A72-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.179312] WDIAG [SQL.RESV] get_table_schema (ob_schema_checker.cpp:889) [46380][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B77-0-0] [lt=28][errcode=-4002] invalid arguments(table_id=18446744073709551615, ret=-4002, lbt()="0x810107f 0x13d10e0d 0x808ce2e 0x14effa7a 0x14fa61b9 0x828d01d 0x81c080b 0x15140419 0x82376fc 0x8235716 0x822a9dd 0x8380932 0x836e7b0 0x1510d393 0x13c52c58 0x7cdaf74 0x7a7b458 0x7a7a129 0x7a767e0 0x78e6cb7 0x78c122b 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.179355] WDIAG [SQL.RESV] get_table_schema (ob_schema_checker.cpp:889) [46380][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B77-0-0] [lt=17][errcode=-4002] invalid arguments(table_id=18446744073709551615, ret=-4002, lbt()="0x810107f 0x13d10e0d 0x808ce2e 0x14effa7a 0x14fa61b9 0x828d01d 0x81c080b 0x15140419 0x82376fc 0x8235716 0x822a9dd 0x8380932 0x836e7b0 0x1510d393 0x13c52c58 0x7cdaf74 0x7a7b458 0x7a7a129 0x7a767e0 0x78e6cb7 0x78c122b 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.179827] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A72-0-0] [lt=14] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580544, proxy_sessid=0, client_sessid=3221580544, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.179909] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A73-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.180682] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A73-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580543, proxy_sessid=0, client_sessid=3221580543, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.180769] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A74-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.181553] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A74-0-0] [lt=11] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580542, proxy_sessid=0, client_sessid=3221580542, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.181641] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A75-0-0] [lt=24] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.182425] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A75-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580541, proxy_sessid=0, client_sessid=3221580541, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.182580] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A76-0-0] [lt=37] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.183419] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A76-0-0] [lt=14] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580540, proxy_sessid=0, client_sessid=3221580540, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.183612] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A77-0-0] [lt=117] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.184552] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A77-0-0] [lt=14] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580539, proxy_sessid=0, client_sessid=3221580539, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.184634] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A78-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:50:57.185824] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A78-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221580538, proxy_sessid=0, client_sessid=3221580538, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:50:57.187881] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=38] ====== check clog disk timer task ====== [2025-01-08 00:50:57.187915] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:50:57.188689] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:57.188744] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=87] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5190487750, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:50:57.189266] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] ====== checkpoint timer task ====== [2025-01-08 00:50:57.189951] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:57.189994] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=42] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:57.190067] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326270087570000, latest_log_ts:{val:1736326253487654001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.190234] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=163] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:50:57.190279] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:57.190297] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] get rec log scn(service_type_=0, rec_log_ts={val:1736326253487654001, v:0}) [2025-01-08 00:50:57.190305] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:50:57.190320] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:57.190340] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:57.190353] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:57.190374] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:57.190396] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:57.190520] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=38] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=471855, clean_num=31457) [2025-01-08 00:50:57.191112] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:57.191166] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=53] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:57.191262] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=73] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251909938001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.191324] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=60] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:50:57.191337] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:50:57.191351] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:57.191365] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:57.191380] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2149797}, scn={val:1736326256925355001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:57.191505] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:50:57.195082] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A447F-0-0] [lt=61][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17933, log_task={header:{begin_lsn:{lsn:2182701}, end_lsn:{lsn:2182823}, log_id:17933, min_scn:{val:1736326256992205000, v:0}, max_scn:{val:1736326256992205000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2182579}, prev_proposal_id:1, committed_end_lsn:{lsn:2182701}, data_checksum:1715074492, accum_checksum:2755098872, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326257090591, freeze_ts:1736326257090591, submit_ts:1736326257090598, flushed_ts:1736326257098753, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:8155}, fs_cb_begin_ts=1736326257195071, log_life_time=104480) [2025-01-08 00:50:57.196091] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285955-0-0] [lt=42] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12276, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257196061, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257196061, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257196061, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.196379] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285955-0-0] [lt=284] success to process renew task(task={tenant_id:1, table_id:12276, add_timestamp:1736326257150429}, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257196061, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257196061, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257196061, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.197656] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18578A-0-0] [lt=11][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.198938] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1736326257195491000, start_id=1736326257195491000, end_id=1736326257195491001) [2025-01-08 00:50:57.199716] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=109] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4551, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:57.199738] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18650704}], scn_array=[cnt:1, {val:1736326256925355002, v:0}], curr_size=122, accum_size=244, time_cost=11563) [2025-01-08 00:50:57.199845] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1561, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:57.199924] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:57.200009] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58360][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83D1-0-0] [lt=27] init point thread id with(&point=0x5572d50e2740, idx_=1016, point=[thread id=58360, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58360) [2025-01-08 00:50:57.202384] INFO [SQL.PC] inner_add_cache_obj (ob_pcv_set.cpp:247) [46380][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B77-0-0] [lt=10] has identical pcv(is_same=true, pcv=0x2ad8cd74a600) [2025-01-08 00:50:57.202471] WDIAG [SQL.PC] add_cache_obj (ob_plan_set.cpp:1200) [46380][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B77-0-0] [lt=67][errcode=-5136] failed to add plan.(ret=-5136) [2025-01-08 00:50:57.202486] WDIAG [SQL.PC] add_cache_obj (ob_i_lib_cache_node.cpp:115) [46380][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B77-0-0] [lt=11][errcode=-5136] failed to inner add cache obj(ret=-5136, key=0x2ad8ad3c9c20, obj=0x2ad8cd730050) [2025-01-08 00:50:57.206042] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46380][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B77-0-0] [lt=9][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.206471] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46380][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B77-0-0] [lt=20][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193762120305, cache_obj->added_lc()=false, cache_obj->get_object_id()=13573, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:57.206807] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18578C-0-0] [lt=51] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11018, add_timestamp:1736326257206790}) [2025-01-08 00:50:57.207215] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18578E-0-0] [lt=27] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11008, add_timestamp:1736326257207212}) [2025-01-08 00:50:57.207535] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=78] ====== traversal_flush timer task ====== [2025-01-08 00:50:57.207637] INFO [MDS] flush (mds_table_mgr.cpp:195) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185392-0-0] [lt=16] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326256925355002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326270087570000, latest_log_ts:{val:1736326253487654001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.207767] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185392-0-0] [lt=96] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251909938001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.207904] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185392-0-0] [lt=56] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251986852000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.208064] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185392-0-0] [lt=126] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251703671004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.208259] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185392-0-0] [lt=191] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:50:57.208652] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46388][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18578F-0-0] [lt=27] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10008, add_timestamp:1736326257208648}) [2025-01-08 00:50:57.212720] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46388][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18578F-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.214021] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46383][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B78-0-0] [lt=21] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11017, add_timestamp:1736326257214015}) [2025-01-08 00:50:57.218116] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B79-0-0] [lt=127][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.219471] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46383][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B78-0-0] [lt=27][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:50:57.219743] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46383][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA3-0-0] [lt=19] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11021, add_timestamp:1736326257219737}) [2025-01-08 00:50:57.222372] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B7A-0-0] [lt=28][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.222576] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B7A-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.222959] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA4-0-0] [lt=16] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12217, add_timestamp:1736326257222953}) [2025-01-08 00:50:57.223003] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA4-0-0] [lt=20] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12142, add_timestamp:1736326257223001}) [2025-01-08 00:50:57.223022] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA4-0-0] [lt=12] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11035, add_timestamp:1736326257223021}) [2025-01-08 00:50:57.223037] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA4-0-0] [lt=8] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12072, add_timestamp:1736326257223036}) [2025-01-08 00:50:57.223050] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA4-0-0] [lt=8] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12068, add_timestamp:1736326257223049}) [2025-01-08 00:50:57.224410] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=41] gc stale ls task succ [2025-01-08 00:50:57.224635] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46502][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA5-0-0] [lt=22][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:10008, add_timestamp:1736326257224630}) [2025-01-08 00:50:57.224660] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46502][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA5-0-0] [lt=26][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:10008, add_timestamp:1736326257224630}) [2025-01-08 00:50:57.224671] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46502][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA5-0-0] [lt=9][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=10008, vtable_locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326242208757, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326242208757, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326242208757, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.226794] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:57.226818] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:57.227187] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46504][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA6-0-0] [lt=39][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326257227184}) [2025-01-08 00:50:57.227301] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46504][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA6-0-0] [lt=112][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326257227184}) [2025-01-08 00:50:57.227309] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46504][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA6-0-0] [lt=8][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=11017, vtable_locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326242209502, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326242209502, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326242209502, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.228556] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46502][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA5-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.230427] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46502][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA8-0-0] [lt=17] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:57.230478] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46502][T1_L0_G0][T1001][YB42C0A89F89-00062B2D1AE84AA8-0-0] [lt=16] get_stable_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:50:57.230506] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46502][T1_L0_G0][T1002][YB42C0A89F89-00062B2D1AE84AA8-0-0] [lt=11] get_stable_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:50:57.233071] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46504][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA6-0-0] [lt=27][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:50:57.233699] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46502][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA8-0-0] [lt=8][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.234671] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.235086] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=26] Slow write(duration=34549, write_len=4096) [2025-01-08 00:50:57.235106] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=17] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8580, offset=2434058}, end_cursor:ObLogCursor{file_id=1, log_id=8581, offset=2434355}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894704050, len:3062, log_data_len:297, seq:8580, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:57.235822] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43069522}], scn_array=[cnt:1, {val:1736326257195491000, v:0}], curr_size=589, accum_size=6007, time_cost=36280) [2025-01-08 00:50:57.236422] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326256925355001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:57.236468] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:57.236492] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:50:57.237062] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50625][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC37D-0-0] [lt=47][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.237284] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:57.237324] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=40] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:57.237377] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251986852000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.237456] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=76] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:50:57.237474] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:50:57.237494] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:57.237510] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:57.237526] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2182823}, scn={val:1736326256992205000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:57.238519] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4015] clock out of order(cur_ts=1736326257238516, cur_ts_=1736326257206471, delta=32045) [2025-01-08 00:50:57.238741] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:57.238827] INFO [PALF] pread (log_reader.cpp:130) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=951) [2025-01-08 00:50:57.239079] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=84] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:336K/78:448145 disk:336K/78:448145 t1m2:68K/15:272233 t1m2g0:68K/15:272233 t1002m2:196K/48:602616 t1002m2g0:196K/48:602616 t1001m2:72K/15:129751 t1001m2g0:72K/15:129751 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:57.239386] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285956-0-0] [lt=100] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11018, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:57.239782] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:50:57.240155] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=196] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8581, offset=2434355}, end_cursor:ObLogCursor{file_id=1, log_id=8582, offset=2434652}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894700050, len:2765, log_data_len:297, seq:8581, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:57.240458] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326256992205000, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:57.240495] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:57.240517] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:50:57.240951] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285956-0-0] [lt=41] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11018, locations=[{table_id:11018, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257240922, sql_renew_time:0, is_mark_fail:false}, {table_id:11018, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257240922, sql_renew_time:0, is_mark_fail:false}, {table_id:11018, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257240922, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.240996] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285956-0-0] [lt=43] success to process renew task(task={tenant_id:1, table_id:11018, add_timestamp:1736326257206790}, locations=[{table_id:11018, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257240922, sql_renew_time:0, is_mark_fail:false}, {table_id:11018, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257240922, sql_renew_time:0, is_mark_fail:false}, {table_id:11018, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257240922, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.241031] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285957-0-0] [lt=25] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:57.241152] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:57.241185] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=33] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:57.241297] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251703671004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.241400] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=100] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:50:57.241418] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:50:57.241437] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:57.241454] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:57.241469] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2172376}, scn={val:1736326256709695004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:57.241606] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=24][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12142, add_timestamp:1736326257241602}) [2025-01-08 00:50:57.241628] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=21][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12142, add_timestamp:1736326257241602}) [2025-01-08 00:50:57.241636] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=12142, vtable_locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326242209644, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.241690] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=32][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12072, add_timestamp:1736326257241688}) [2025-01-08 00:50:57.241699] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=9][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12072, add_timestamp:1736326257241688}) [2025-01-08 00:50:57.241706] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=12072, vtable_locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326242209730, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.241726] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=12] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12084, add_timestamp:1736326257241723}) [2025-01-08 00:50:57.241740] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12068, add_timestamp:1736326257241740}) [2025-01-08 00:50:57.241747] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12068, add_timestamp:1736326257241740}) [2025-01-08 00:50:57.241753] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=5][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=12068, vtable_locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326242209765, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.241810] INFO [PALF] pread (log_reader.cpp:130) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=722) [2025-01-08 00:50:57.243498] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0deadb0, session_id:3221580540, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:205}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.243544] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=37] kill and revert session(conn.sessid_=3221580540, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.243554] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0de8030, session_id:3221580538, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.243571] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221580538, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.243577] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dea030, session_id:3221580539, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.243592] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221580539, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.243599] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8a0deadb0, session_id:3221580540, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:205}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.243642] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221580540, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.243654] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221580540, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B7A-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.243692] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=35] sock ref clean, do destroy(*s={this:0x2ad8a0de8030, session_id:3221580538, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.243724] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221580538, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.243733] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221580538, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B78-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:241, rec_:241, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.243759] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] sock ref clean, do destroy(*s={this:0x2ad8a0dea030, session_id:3221580539, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.243788] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221580539, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.243796] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221580539, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B79-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:633, rec_:633, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.243923] INFO [SERVER] run (obmp_disconnect.cpp:83) [46503][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=25] free session successfully(sessid=3221580540) [2025-01-08 00:50:57.243944] INFO [SERVER] run (obmp_disconnect.cpp:90) [46503][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=21] mark session id unused(sessid=3221580540) [2025-01-08 00:50:57.243990] INFO [SERVER] run (obmp_disconnect.cpp:83) [46503][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221580538) [2025-01-08 00:50:57.244000] INFO [SERVER] run (obmp_disconnect.cpp:90) [46503][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221580538) [2025-01-08 00:50:57.244054] INFO [SERVER] run (obmp_disconnect.cpp:83) [46503][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221580539) [2025-01-08 00:50:57.244064] INFO [SERVER] run (obmp_disconnect.cpp:90) [46503][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221580539) [2025-01-08 00:50:57.244202] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46503][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCD-0-0] [lt=5] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11037, add_timestamp:1736326257244198}) [2025-01-08 00:50:57.248291] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46503][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCE-0-0] [lt=17] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12325, add_timestamp:1736326257248287}) [2025-01-08 00:50:57.248401] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46503][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCE-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326257206471000, start_id=1736326257206471000, end_id=1736326257206471001) [2025-01-08 00:50:57.251886] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46508][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD0-0-0] [lt=22] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11003, add_timestamp:1736326257251880}) [2025-01-08 00:50:57.252800] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46503][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCE-0-0] [lt=17][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:50:57.253057] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58332][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=22] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:50:57.253864] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffc030, session_id:3221580531, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.253909] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=36] kill and revert session(conn.sessid_=3221580531, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.253920] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffcdb0, session_id:3221580532, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.253937] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] kill and revert session(conn.sessid_=3221580532, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.253944] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8db0, session_id:3221580529, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:291, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.253959] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221580529, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.253966] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8efffc030, session_id:3221580531, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.254006] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221580531, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.254018] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221580531, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B18578E-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:133, rec_:133, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:31, seq_:37}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:38}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.254058] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=37] sock ref clean, do destroy(*s={this:0x2ad8efffcdb0, session_id:3221580532, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.254087] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221580532, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.254095] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221580532, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B18578F-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:436, rec_:436, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.254122] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] sock ref clean, do destroy(*s={this:0x2ad8efff8db0, session_id:3221580529, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:291, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.254150] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221580529, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.254157] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221580529, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B18578C-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:186, rec_:186, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:5, seq_:7}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.254351] INFO [SERVER] run (obmp_disconnect.cpp:83) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] free session successfully(sessid=3221580531) [2025-01-08 00:50:57.254371] INFO [SERVER] run (obmp_disconnect.cpp:90) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=19] mark session id unused(sessid=3221580531) [2025-01-08 00:50:57.254425] INFO [SERVER] run (obmp_disconnect.cpp:83) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=7] free session successfully(sessid=3221580532) [2025-01-08 00:50:57.254435] INFO [SERVER] run (obmp_disconnect.cpp:90) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221580532) [2025-01-08 00:50:57.254479] INFO [SERVER] run (obmp_disconnect.cpp:83) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221580529) [2025-01-08 00:50:57.254490] INFO [SERVER] run (obmp_disconnect.cpp:90) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221580529) [2025-01-08 00:50:57.255154] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0df0030, session_id:3221580546, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:210}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.255187] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221580546, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.255273] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=83][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dee030, session_id:3221580544, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:208}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.255296] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221580544, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.255303] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0deedb0, session_id:3221580545, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:209}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.255318] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221580545, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.255325] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0decdb0, session_id:3221580543, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:207}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.255340] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] kill and revert session(conn.sessid_=3221580543, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.255346] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8a0df0030, session_id:3221580546, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:210}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.255398] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221580546, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.255411] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221580546, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AA8-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:562, rec_:562, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.255446] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] sock ref clean, do destroy(*s={this:0x2ad8a0dee030, session_id:3221580544, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:208}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.255480] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221580544, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.255488] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221580544, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AA6-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:549, rec_:549, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:12, seq_:19}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:20}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.255514] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] sock ref clean, do destroy(*s={this:0x2ad8a0deedb0, session_id:3221580545, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:209}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.255588] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221580545, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.255601] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221580545, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AA7-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:368, rec_:368, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.255630] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] sock ref clean, do destroy(*s={this:0x2ad8a0decdb0, session_id:3221580543, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:207}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.255663] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221580543, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.255671] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221580543, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AA5-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.255784] INFO [SERVER] run (obmp_disconnect.cpp:83) [46499][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=36] free session successfully(sessid=3221580546) [2025-01-08 00:50:57.255804] INFO [SERVER] run (obmp_disconnect.cpp:90) [46499][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] mark session id unused(sessid=3221580546) [2025-01-08 00:50:57.255849] INFO [SERVER] run (obmp_disconnect.cpp:83) [46499][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221580544) [2025-01-08 00:50:57.255859] INFO [SERVER] run (obmp_disconnect.cpp:90) [46499][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221580544) [2025-01-08 00:50:57.255901] INFO [SERVER] run (obmp_disconnect.cpp:83) [46499][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221580545) [2025-01-08 00:50:57.255910] INFO [SERVER] run (obmp_disconnect.cpp:90) [46499][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221580545) [2025-01-08 00:50:57.255951] INFO [SERVER] run (obmp_disconnect.cpp:83) [46499][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=4] free session successfully(sessid=3221580543) [2025-01-08 00:50:57.255960] INFO [SERVER] run (obmp_disconnect.cpp:90) [46499][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221580543) [2025-01-08 00:50:57.256820] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=81] Slow write(duration=14954, write_len=4096) [2025-01-08 00:50:57.256842] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=21] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8582, offset=2434652}, end_cursor:ObLogCursor{file_id=1, log_id=8583, offset=2434949}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894708050, len:2468, log_data_len:297, seq:8582, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:57.257010] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326256709695004, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:57.257052] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:57.257074] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:50:57.257084] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:50:57.260986] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=21] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:57.261006] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=20] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:57.261072] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8497E-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:57.261093] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8497E-0-0] [lt=18] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:57.261992] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285957-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11008, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257261915, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257261915, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257261915, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.262033] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285957-0-0] [lt=40] success to process renew task(task={tenant_id:1, table_id:11008, add_timestamp:1736326257207212}, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257261915, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257261915, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257261915, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.262074] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285958-0-0] [lt=26] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=10008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:57.263385] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C295-0-0] [lt=50] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:57.263424] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C295-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326256992205000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326256992205000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:57.264435] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285958-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10008, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257264420, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257264420, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257264420, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.264551] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285958-0-0] [lt=114] success to process renew task(task={tenant_id:1, table_id:10008, add_timestamp:1736326257208648}, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257264420, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257264420, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257264420, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.264590] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285959-0-0] [lt=30] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11017, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:57.266302] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285959-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11017, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266289, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266289, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266289, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.266338] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285959-0-0] [lt=35] success to process renew task(task={tenant_id:1, table_id:11017, add_timestamp:1736326257214015}, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266289, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266289, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266289, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.266370] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595A-0-0] [lt=24] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11021, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:57.266711] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595A-0-0] [lt=9] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11021, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266700, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266700, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266700, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.266742] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595A-0-0] [lt=30] success to process renew task(task={tenant_id:1, table_id:11021, add_timestamp:1736326257219737}, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266700, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266700, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266700, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.266778] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595B-0-0] [lt=23] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12217, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266772, sql_renew_time:1736326257266772, is_mark_fail:false}]) [2025-01-08 00:50:57.266792] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595B-0-0] [lt=14] success to process renew task(task={tenant_id:1, table_id:12217, add_timestamp:1736326257222953}, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266772, sql_renew_time:1736326257266772, is_mark_fail:false}]) [2025-01-08 00:50:57.266869] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595C-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12142, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266808, sql_renew_time:1736326257266808, is_mark_fail:false}]) [2025-01-08 00:50:57.266889] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595C-0-0] [lt=20] success to process renew task(task={tenant_id:1, table_id:12142, add_timestamp:1736326257223001}, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266808, sql_renew_time:1736326257266808, is_mark_fail:false}]) [2025-01-08 00:50:57.266913] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595D-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11035, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266908, sql_renew_time:1736326257266908, is_mark_fail:false}]) [2025-01-08 00:50:57.266927] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595D-0-0] [lt=14] success to process renew task(task={tenant_id:1, table_id:11035, add_timestamp:1736326257223021}, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266908, sql_renew_time:1736326257266908, is_mark_fail:false}]) [2025-01-08 00:50:57.266947] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595E-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12072, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266943, sql_renew_time:1736326257266943, is_mark_fail:false}]) [2025-01-08 00:50:57.266960] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595E-0-0] [lt=13] success to process renew task(task={tenant_id:1, table_id:12072, add_timestamp:1736326257223036}, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266943, sql_renew_time:1736326257266943, is_mark_fail:false}]) [2025-01-08 00:50:57.266979] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595F-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12068, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266975, sql_renew_time:1736326257266975, is_mark_fail:false}]) [2025-01-08 00:50:57.266992] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28595F-0-0] [lt=13] success to process renew task(task={tenant_id:1, table_id:12068, add_timestamp:1736326257223049}, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266975, sql_renew_time:1736326257266975, is_mark_fail:false}]) [2025-01-08 00:50:57.267011] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285960-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12084, locations=[{table_id:12084, partition_id:12084, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257267007, sql_renew_time:1736326257267007, is_mark_fail:false}]) [2025-01-08 00:50:57.267024] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285960-0-0] [lt=13] success to process renew task(task={tenant_id:1, table_id:12084, add_timestamp:1736326257241723}, locations=[{table_id:12084, partition_id:12084, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257267007, sql_renew_time:1736326257267007, is_mark_fail:false}]) [2025-01-08 00:50:57.267039] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285961-0-0] [lt=11] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11037, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:57.267395] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87034-0-0] [lt=9][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.267444] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:57.267456] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:57.267622] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285961-0-0] [lt=8] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11037, locations=[{table_id:11037, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257267608, sql_renew_time:0, is_mark_fail:false}, {table_id:11037, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257267608, sql_renew_time:0, is_mark_fail:false}, {table_id:11037, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257267608, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.267653] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285961-0-0] [lt=31] success to process renew task(task={tenant_id:1, table_id:11037, add_timestamp:1736326257244198}, locations=[{table_id:11037, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257267608, sql_renew_time:0, is_mark_fail:false}, {table_id:11037, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257267608, sql_renew_time:0, is_mark_fail:false}, {table_id:11037, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257267608, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.267683] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285962-0-0] [lt=22] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12325, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:57.268012] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=57] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=8801) [2025-01-08 00:50:57.268490] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe4c030, session_id:3221580537, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.268534] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=37] kill and revert session(conn.sessid_=3221580537, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.268547] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] sock ref clean, do destroy(*s={this:0x2ad8efe4c030, session_id:3221580537, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.268641] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221580537, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.268657] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221580537, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CD0-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:98, rec_:98, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:10}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:11}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.268768] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe4adb0, session_id:3221580536, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.268795] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221580536, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.268803] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sock ref clean, do destroy(*s={this:0x2ad8efe4adb0, session_id:3221580536, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.268848] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221580536, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.268860] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221580536, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CCF-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:134, rec_:134, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:7}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.268894] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe48db0, session_id:3221580534, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:292, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.268912] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221580534, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.268919] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8efe48db0, session_id:3221580534, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:292, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.268976] INFO [SERVER] run (obmp_disconnect.cpp:83) [46502][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=18] free session successfully(sessid=3221580537) [2025-01-08 00:50:57.268999] INFO [SERVER] run (obmp_disconnect.cpp:90) [46502][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=22] mark session id unused(sessid=3221580537) [2025-01-08 00:50:57.269011] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221580534, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.269025] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221580534, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CCD-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:187, rec_:187, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:2, seq_:4}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:5}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:6}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.269050] INFO [SERVER] run (obmp_disconnect.cpp:83) [46502][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221580536) [2025-01-08 00:50:57.269061] INFO [SERVER] run (obmp_disconnect.cpp:90) [46502][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=11] mark session id unused(sessid=3221580536) [2025-01-08 00:50:57.269103] INFO [SERVER] run (obmp_disconnect.cpp:83) [46502][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221580534) [2025-01-08 00:50:57.269113] INFO [SERVER] run (obmp_disconnect.cpp:90) [46502][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221580534) [2025-01-08 00:50:57.270510] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285962-0-0] [lt=9] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12325, locations=[{table_id:12325, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270493, sql_renew_time:0, is_mark_fail:false}, {table_id:12325, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270493, sql_renew_time:0, is_mark_fail:false}, {table_id:12325, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270493, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.270551] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285962-0-0] [lt=40] success to process renew task(task={tenant_id:1, table_id:12325, add_timestamp:1736326257248287}, locations=[{table_id:12325, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270493, sql_renew_time:0, is_mark_fail:false}, {table_id:12325, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270493, sql_renew_time:0, is_mark_fail:false}, {table_id:12325, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270493, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.270642] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285963-0-0] [lt=79] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11003, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:57.270942] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285963-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11003, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270929, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270929, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270929, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.270977] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285963-0-0] [lt=34] success to process renew task(task={tenant_id:1, table_id:11003, add_timestamp:1736326257251880}, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270929, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270929, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257270929, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:57.276677] INFO [SQL.PC] inner_add_cache_obj (ob_pcv_set.cpp:247) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA2-0-0] [lt=11] has identical pcv(is_same=true, pcv=0x2ad8cd74a600) [2025-01-08 00:50:57.276735] WDIAG [SQL.PC] add_cache_obj (ob_plan_set.cpp:1200) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA2-0-0] [lt=39][errcode=-5136] failed to add plan.(ret=-5136) [2025-01-08 00:50:57.276751] WDIAG [SQL.PC] add_cache_obj (ob_i_lib_cache_node.cpp:115) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA2-0-0] [lt=12][errcode=-5136] failed to inner add cache obj(ret=-5136, key=0x2ad8b784fc20, obj=0x2ad8cd6d8050) [2025-01-08 00:50:57.277147] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=11][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277172] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=24][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277197] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=13][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277205] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277219] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277226] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277238] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277244] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277373] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277391] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=18][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277411] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=12][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277465] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=54][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277487] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=10][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277495] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277508] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277515] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277528] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277534] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277547] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277553] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277566] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277572] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277585] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277591] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277603] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277610] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277622] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277629] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277641] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277647] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277659] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277666] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277678] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277684] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277696] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277731] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=34][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277749] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=9][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277756] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277769] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277776] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277789] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277795] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277807] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277814] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277827] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277833] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277820] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=30] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5190514884, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:50:57.277846] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277852] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277864] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277871] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277883] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277889] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277901] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277908] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277920] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277927] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277939] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277945] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277957] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277964] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.277976] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.277982] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278084] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=89][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278096] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=13][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278111] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=9][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278118] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278130] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278137] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278149] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278156] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278168] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278174] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278261] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=74][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278269] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=9][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278283] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278289] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278302] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278308] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278321] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278327] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278387] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=48][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278412] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=25][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278430] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=9][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278438] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278450] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278457] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278470] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278476] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278489] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278495] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278508] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278514] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278527] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278533] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278545] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278552] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278564] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278570] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278582] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278589] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278601] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278608] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278620] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278626] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278638] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278645] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278657] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278664] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278676] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278682] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278694] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278701] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278713] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278719] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278731] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278738] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278750] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278756] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278812] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278824] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=12][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278841] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278847] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278860] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278867] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278879] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278886] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278898] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278904] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278917] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278923] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278935] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:50:57.278942] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:50:57.278954] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCC-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:50:57.280272] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:57.282530] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=12][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:57.282908] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=188] gc stale ls task succ [2025-01-08 00:50:57.282977] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:57.282996] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-01-08 00:50:57.283005] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:57.283018] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:57.283030] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193761196428, flush_io_finished_ret_:0, flush_io_finished_round_:1920, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:57.290794] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=46] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:57.291343] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2149797}], scn_array=[cnt:1, {val:1736326256925355002, v:0}], curr_size=122, accum_size=244, time_cost=103169) [2025-01-08 00:50:57.291651] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=17663, log_task={header:{begin_lsn:{lsn:2149797}, end_lsn:{lsn:2149919}, log_id:17663, min_scn:{val:1736326256925355002, v:0}, max_scn:{val:1736326256925355002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2149797}, data_checksum:2222604892, accum_checksum:3352407196, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326257174786, freeze_ts:1736326257174789, submit_ts:1736326257174819, flushed_ts:1736326257291615, gen_to_freeze cost time:3, gen_to_submit cost time:33, submit_to_flush cost time:116796}, fs_cb_begin_ts=1736326257291628, log_life_time=116842) [2025-01-08 00:50:57.295092] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9F-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:57.295140] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C9F-0-0] [lt=44] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:57.311770] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_stat.cpp:301) [46383][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA3-0-0] [lt=20][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:50:57.314073] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185414-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.314506] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=91][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18650826}], scn_array=[cnt:1, {val:1736326256925355003, v:0}], curr_size=122, accum_size=366, time_cost=26359) [2025-01-08 00:50:57.316054] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:50:57.316097] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221580530, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.316109] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] sock ref clean, do destroy(*s={this:0x2ad8efffadb0, session_id:3221580530, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:617, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.316144] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221580530, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.316155] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221580530, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B18578D-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:512, rec_:512, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.316387] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=19] free session successfully(sessid=3221580530) [2025-01-08 00:50:57.316406] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=18] mark session id unused(sessid=3221580530) [2025-01-08 00:50:57.317820] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:50:57.317861] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221580541, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.317872] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sock ref clean, do destroy(*s={this:0x2ad8a0de8db0, session_id:3221580541, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.317946] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221580541, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.317959] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221580541, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AA3-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:688, rec_:688, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:219, seq_:225}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:226}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.318305] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=16] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8583, offset=2434949}, end_cursor:ObLogCursor{file_id=1, log_id=8584, offset=2435254}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89470c050, len:2171, log_data_len:305, seq:8583, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:57.318647] INFO [SERVER] run (obmp_disconnect.cpp:83) [46390][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=39] free session successfully(sessid=3221580541) [2025-01-08 00:50:57.318711] INFO [SERVER] run (obmp_disconnect.cpp:90) [46390][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=62] mark session id unused(sessid=3221580541) [2025-01-08 00:50:57.319140] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E42-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:57.319229] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E42-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326256925355003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326256925355003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:57.319329] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.319359] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.320156] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBE-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326257423792, keepalive_ts:-1}) [2025-01-08 00:50:57.320274] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBE-0-0] [lt=115] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326257423792, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:57.320921] INFO [LIB] stat (utility.h:1210) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C295-0-0] [lt=40] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:57.325319] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E42-0-0] [lt=21] get number(ret=0, service_type_=0, range=1, base_id=1736326257321437000, start_id=1736326257321437000, end_id=1736326257321437001) [2025-01-08 00:50:57.327963] INFO [SQL.PC] inner_add_cache_obj (ob_pcv_set.cpp:247) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185789-0-0] [lt=10] has identical pcv(is_same=true, pcv=0x2ad8cd74a600) [2025-01-08 00:50:57.328034] WDIAG [SQL.PC] add_cache_obj (ob_plan_set.cpp:1200) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185789-0-0] [lt=49][errcode=-5136] failed to add plan.(ret=-5136) [2025-01-08 00:50:57.328051] WDIAG [SQL.PC] add_cache_obj (ob_i_lib_cache_node.cpp:115) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185789-0-0] [lt=12][errcode=-5136] failed to inner add cache obj(ret=-5136, key=0x2ad8b79c9c20, obj=0x2ad966bd4050) [2025-01-08 00:50:57.331903] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA2-0-0] [lt=9][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.332415] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA2-0-0] [lt=32][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:50:57.333154] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43070111}], scn_array=[cnt:1, {val:1736326257320355000, v:0}], curr_size=750, accum_size=6757, time_cost=12125) [2025-01-08 00:50:57.333633] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185789-0-0] [lt=9][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.334097] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185789-0-0] [lt=22][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:50:57.335166] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:57.335269] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=99][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.345578] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=36][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:57.346244] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=26] current monitor number(seq_=-1) [2025-01-08 00:50:57.346908] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58351][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3859-0-0] [lt=24] init point thread id with(&point=0x5572d50e2500, idx_=1007, point=[thread id=58351, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58351) [2025-01-08 00:50:57.347285] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=134][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43070861}], scn_array=[cnt:1, {val:1736326257321437000, v:0}], curr_size=1135, accum_size=7892, time_cost=12162) [2025-01-08 00:50:57.348822] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=41] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8584, offset=2435254}, end_cursor:ObLogCursor{file_id=1, log_id=8585, offset=2435551}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89471c050, len:1866, log_data_len:297, seq:8584, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:57.349463] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E42-0-0] [lt=47] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:57.349509] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E42-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326256925355002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326256925355002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:57.369317] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43072585}], scn_array=[cnt:1, {val:1736326257349471000, v:0}], curr_size=750, accum_size=9231, time_cost=16245) [2025-01-08 00:50:57.379314] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=20] gc stale ls task succ [2025-01-08 00:50:57.379563] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3747-0-0] [lt=27] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=16, avg_log_submit_to_flush_time=7961, avg_log_submit_to_slide_time=60291) [2025-01-08 00:50:57.379600] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3747-0-0] [lt=35][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17849, log_task={header:{begin_lsn:{lsn:2172498}, end_lsn:{lsn:2172620}, log_id:17849, min_scn:{val:1736326256709695006, v:0}, max_scn:{val:1736326256709695006, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2172376}, prev_proposal_id:1, committed_end_lsn:{lsn:2172498}, data_checksum:940009244, accum_checksum:2734137090, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326257261787, freeze_ts:1736326257261787, submit_ts:1736326257261800, flushed_ts:1736326257265505, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:3705}, fs_cb_begin_ts=1736326257379553, log_life_time=117766) [2025-01-08 00:50:57.379759] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:57.379852] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17760, miss_cnt:0, cache_read_size:2172620, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:57.386627] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=146] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:50:57.386710] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=82] try gc retain ctx [2025-01-08 00:50:57.386722] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-01-08 00:50:57.386753] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326256925355003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326246900070004, v:0}, keep_alive_scn_:{val:1736326251909938004, v:0}, update_ts_:1736326257372912}) [2025-01-08 00:50:57.387466] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326254917395002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326270087570000, latest_log_ts:{val:1736326253487654001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:57.387619] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=79] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=55, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:57.387680] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] finish update min start scn(max_decided_scn={val:1736326256925355002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326246900070004, v:0}, keep_alive_scn_:{val:1736326251909938004, v:0}, update_ts_:1736326257372912}) [2025-01-08 00:50:57.388467] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326256925355001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326254917395002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:57.388610] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=63] finish update min start scn(max_decided_scn={val:1736326256992205002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326249981471000, v:0}, keep_alive_scn_:{val:1736326254991003000, v:0}, update_ts_:1736326257372912}) [2025-01-08 00:50:57.389243] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326256992205000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:57.389504] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=247] finish update min start scn(max_decided_scn={val:1736326256709695006, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326248201055002, v:0}, keep_alive_scn_:{val:1736326253203138002, v:0}, update_ts_:1736326257372912}) [2025-01-08 00:50:57.390296] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326256709695004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326253912776002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:57.392460] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:57.392491] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-01-08 00:50:57.392501] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:57.392515] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:57.392529] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193761306156, flush_io_finished_ret_:0, flush_io_finished_round_:1916, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:57.393986] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=22] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:57.394313] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6716485632, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:57.394588] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=15] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:57.397269] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:50:57.397315] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38] kill and revert session(conn.sessid_=3221580533, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.397497] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=180] sock ref clean, do destroy(*s={this:0x2ad8efe48030, session_id:3221580533, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.397552] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] unlock session of tenant(conn.sessid_=3221580533, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.397566] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221580533, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CCC-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:123, rec_:123, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:1, seq_:3}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:4}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:5}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.398967] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AC48-0-0] [lt=29] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.743243837356567383e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=387, Throttle Trigger(MB)=232, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.808276397751752711e+02) [2025-01-08 00:50:57.399589] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Slow write(duration=19033, write_len=4096) [2025-01-08 00:50:57.399612] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=21] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8585, offset=2435551}, end_cursor:ObLogCursor{file_id=1, log_id=8586, offset=2435848}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894714050, len:1569, log_data_len:297, seq:8585, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:57.399674] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:57.399699] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-01-08 00:50:57.399706] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:57.399716] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:57.399727] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193761313166, flush_io_finished_ret_:0, flush_io_finished_round_:3744, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:57.399921] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=1] free session successfully(sessid=3221580533) [2025-01-08 00:50:57.399957] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=34] mark session id unused(sessid=3221580533) [2025-01-08 00:50:57.402574] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=17] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8586, offset=2435848}, end_cursor:ObLogCursor{file_id=1, log_id=8587, offset=2436145}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894710050, len:1272, log_data_len:297, seq:8586, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:57.402802] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=47] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=6375, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:57.402827] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9820) [2025-01-08 00:50:57.402844] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=248, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:57.402871] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=122] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=47, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:57.403932] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=30] ====== tenant freeze timer task ====== [2025-01-08 00:50:57.404472] INFO [LIB] stat (utility.h:1210) [58356][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA385C-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:57.404496] INFO [LIB] stat (utility.h:1210) [58356][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA385C-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=9964, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:57.404507] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58356][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA385C-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=1382, avg_log_gen_to_submit_time=1463, avg_log_submit_to_flush_time=6707, avg_log_submit_to_slide_time=8501) [2025-01-08 00:50:57.406901] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9820, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:57.407065] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=55] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326256709695004, v:0}, new_max_version={val:1736326256709695004, v:0}, last_min_version={val:1736326256491846004, v:0}, last_max_version={val:1736326256491846004, v:0}, total_time_us=9591, delta_us=697367) [2025-01-08 00:50:57.407094] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326256709695004, v:0}, min_version={val:1736326256709695004, v:0}, max_version={val:1736326256709695004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:57.407114] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:57.414540] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2] kill and revert session(conn.sessid_=3221580542, proxy_sessid=0, ret=0) [2025-01-08 00:50:57.414573] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] sock ref clean, do destroy(*s={this:0x2ad8a0dec030, session_id:3221580542, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:206}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.414623] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] unlock session of tenant(conn.sessid_=3221580542, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:57.414636] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221580542, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AA4-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:119, rec_:119, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:2, seq_:4}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:5}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.414894] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] free session successfully(sessid=3221580542) [2025-01-08 00:50:57.414917] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=22] mark session id unused(sessid=3221580542) [2025-01-08 00:50:57.416939] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=22] ====== traversal_flush timer task ====== [2025-01-08 00:50:57.417100] INFO [MDS] flush (mds_table_mgr.cpp:195) [50558][T1001_TimerWK0][T1001][YB42C0A89F89-00062B2D1E484E10-0-0] [lt=27] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326257349471000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269284252000, latest_log_ts:{val:1736326252703023001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.417280] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50558][T1001_TimerWK0][T1001][YB42C0A89F89-00062B2D1E484E10-0-0] [lt=173] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:50:57.425010] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:50:57.425064] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=48] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:50:57.425077] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:50:57.425089] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:57.427801] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=130, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}]) [2025-01-08 00:50:57.428120] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=313] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:57.445350] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.459461] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.468586] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=48] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:57.468848] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.471731] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=6230, avg_group_log_size=479) [2025-01-08 00:50:57.471777] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=6230) [2025-01-08 00:50:57.471797] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:57.472490] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:57.483775] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2172742}], scn_array=[cnt:1, {val:1736326257262115001, v:0}], curr_size=122, accum_size=488, time_cost=20243) [2025-01-08 00:50:57.491232] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84207921}], scn_array=[cnt:1, {val:1736326257422047000, v:0}], curr_size=122, accum_size=4269, time_cost=19267) [2025-01-08 00:50:57.491580] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4485-0-0] [lt=29] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=15, avg_log_submit_to_flush_time=6694, avg_log_submit_to_slide_time=43016) [2025-01-08 00:50:57.492315] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:57.495021] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-01-08 00:50:57.495063] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=36] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70285300, tenant_id=1001) [2025-01-08 00:50:57.495087] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:57.520340] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.537369] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1736326257537365, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008717, add_bytes: 0 [2025-01-08 00:50:57.542273] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=88] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.544306] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.546391] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.566460] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43073924}], scn_array=[cnt:1, {val:1736326257533294000, v:0}], curr_size=589, accum_size=589, time_cost=29517) [2025-01-08 00:50:57.575888] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:57.586246] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=62, size_used=0, mem_used=8318976) [2025-01-08 00:50:57.587120] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2172864}], scn_array=[cnt:1, {val:1736326257262115002, v:0}], curr_size=122, accum_size=610, time_cost=24787) [2025-01-08 00:50:57.589394] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=34] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:57.590605] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:57.590635] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1089, avg_group_log_size=121) [2025-01-08 00:50:57.590652] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1089) [2025-01-08 00:50:57.590666] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:57.592573] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:57.593870] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58245][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=24] worker thread exit(tenant_->id()=1002, workers_.get_size()=5) [2025-01-08 00:50:57.594368] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9023-0-0] [lt=51] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1089) [2025-01-08 00:50:57.594949] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:57.595994] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=11] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1116688, replace_num=15728) [2025-01-08 00:50:57.596020] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:57.608665] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2183311}], scn_array=[cnt:1, {val:1736326257495620000, v:0}], curr_size=122, accum_size=854, time_cost=21854) [2025-01-08 00:50:57.608915] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18651183}], scn_array=[cnt:1, {val:1736326257422047001, v:0}], curr_size=122, accum_size=723, time_cost=18142) [2025-01-08 00:50:57.609056] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:57.609078] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12659, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:57.609088] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=77, avg_log_submit_to_flush_time=11613, avg_log_submit_to_slide_time=12581) [2025-01-08 00:50:57.609318] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2150154}], scn_array=[cnt:1, {val:1736326257422047001, v:0}], curr_size=122, accum_size=601, time_cost=18568) [2025-01-08 00:50:57.609390] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:57.609404] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=19747, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:57.612903] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EB-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:57.612947] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EB-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326257262115001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326257262115001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:57.633741] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43074513}], scn_array=[cnt:1, {val:1736326257597253000, v:0}], curr_size=589, accum_size=1178, time_cost=31726) [2025-01-08 00:50:57.646473] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.659576] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:57.660274] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=60][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43075102}], scn_array=[cnt:1, {val:1736326257638161000, v:0}], curr_size=750, accum_size=1928, time_cost=19542) [2025-01-08 00:50:57.660495] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=36] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:57.672007] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:57.672043] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=98) [2025-01-08 00:50:57.681327] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5191417137, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:50:57.681364] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:57.681433] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=68] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a1d10) [2025-01-08 00:50:57.681443] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5191417137, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:50:57.685582] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4489-0-0] [lt=34] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:57.688452] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=53][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84208629}], scn_array=[cnt:1, {val:1736326257492875001, v:0}], curr_size=122, accum_size=4977, time_cost=16515) [2025-01-08 00:50:57.698377] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2154372, this={dio_aligned_buf:{buf_write_offset:4094, buf_padding_size:2, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:22440, truncate_used_ts:5052}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2150400, buf_len=122, total_write_size=12172, total_write_size_after_dio=417792, ob_pwrite_used_ts=393486, count=100, each_pwrite_cost=3934) [2025-01-08 00:50:57.698422] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=17202, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:57.698439] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2151, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:57.701935] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085147-0-0] [lt=11] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:57.729625] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=102] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:50:57.745370] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] get_max_applied_scn(scn={val:1736326257597253000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43076441}, palf_committed_end_scn_:{val:1736326257734840000, v:0}, last_check_scn_:{val:1736326257734840000, v:0}, max_applied_cb_scn_:{val:1736326257597253000, v:0}}) [2025-01-08 00:50:57.745424] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=48] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43076441}, readable_log_tail:{lsn:43076441}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:24028327, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:57.746594] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.762457] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=42] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:57.762743] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC374F-0-0] [lt=29] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:57.762779] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC374F-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:57.766282] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D69A-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:57.766310] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D69A-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=64938, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:57.767949] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:57.780277] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=79] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:57.790520] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D69B-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=36, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:57.790548] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D69B-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=46936, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:57.795610] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:57.796665] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:50:57.799686] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D017-0-0] [lt=64][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.802376] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9820, avg_group_log_size=654) [2025-01-08 00:50:57.802404] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9820) [2025-01-08 00:50:57.823263] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=154] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:57.846723] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.851747] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=14] ====== traversal_flush timer task ====== [2025-01-08 00:50:57.851961] INFO [MDS] flush (mds_table_mgr.cpp:195) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8575A-0-0] [lt=82] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326257690273000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:57.852070] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8575A-0-0] [lt=103] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:50:57.858251] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1002}, {id:1001}, {id:1}]) [2025-01-08 00:50:57.872678] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=62] PNIO [ratelimit] time: 1736326257872543, bytes: 868247881, bw: 0.225023 MB/s, add_ts: 1001234, add_bytes: 236245 [2025-01-08 00:50:57.873491] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=5629, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:57.873572] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=79] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5685) [2025-01-08 00:50:57.873593] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1335, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:57.873680] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=44, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:57.875383] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5AF2-0-0] [lt=43] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:57.875408] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5AF2-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=8978, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:57.875419] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5AF2-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=16, avg_log_gen_to_submit_time=95, avg_log_submit_to_flush_time=7029, avg_log_submit_to_slide_time=8883) [2025-01-08 00:50:57.875521] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5685, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:57.887336] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A448D-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:57.887429] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A448D-0-0] [lt=88] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:57.890716] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=204] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6189, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:57.890745] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:57.890765] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=284, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:57.891321] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=6] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:57.897302] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.897343] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.897367] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.905925] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=46] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:50:57.906042] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:57.906434] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18651549}], scn_array=[cnt:1, {val:1736326257422047004, v:0}], curr_size=122, accum_size=1089, time_cost=15282) [2025-01-08 00:50:57.908427] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2150520}], scn_array=[cnt:1, {val:1736326257422047004, v:0}], curr_size=122, accum_size=967, time_cost=17245) [2025-01-08 00:50:57.909584] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84209459}], scn_array=[cnt:1, {val:1736326257891920000, v:0}], curr_size=586, accum_size=586, time_cost=15165) [2025-01-08 00:50:57.915189] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:57.916769] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=345] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1736326257690273000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1736326257916676}, version_delta=226494) [2025-01-08 00:50:57.929023] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=74] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:57.938787] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10409) [2025-01-08 00:50:57.946702] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] kill and revert session(conn.sessid_=3221580517, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:50:57.946863] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=104] sock ref clean, do destroy(*s={this:0x2ad8a0dfc030, session_id:3221580517, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326257080522, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:57.946982] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=81] unlock session of tenant(conn.sessid_=3221580517, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:50:57.946999] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221580517, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AA1-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:920081, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:920082, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:920083, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:920083, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:920083, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:920083, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:920083, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:920083, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:920083, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:57.947292] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=87][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:57.947416] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=7] free session successfully(sessid=3221580517) [2025-01-08 00:50:57.947454] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=36] mark session id unused(sessid=3221580517) [2025-01-08 00:50:57.965076] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C89-0-0] [lt=56][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.966501] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C89-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:57.969421] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:57.972682] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:57.994441] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:57.996353] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:57.996609] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:50:57.996831] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:50:57.996886] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:50:57.997329] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1089) [2025-01-08 00:50:58.048302] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:58.048438] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=134][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.070000] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=541, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:58.070042] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=41] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:58.070361] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185116-0-0] [lt=47] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:50:58.076304] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:58.077404] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=59, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:58.077558] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:58.077606] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:12, bw:55410, [delay/us]:prepare:7, schedule:341, submit:1032, rt:5296, total:6678) [2025-01-08 00:50:58.077621] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:58.077677] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:12, avg_bw:55413, [delay/us]: prepare:7, schedule:341, submit:1032, device:5296, total:6678) [2025-01-08 00:50:58.077690] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:58.077728] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4973, iops:14, bw:67288, [delay/us]:prepare:10, schedule:690, submit:849, rt:10363, total:11914) [2025-01-08 00:50:58.077738] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:58.077772] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4974, avg_iops:14, avg_bw:67288, [delay/us]: prepare:10, schedule:690, submit:849, device:10363, total:11914) [2025-01-08 00:50:58.077783] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:58.077804] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:7, bw:27708, [delay/us]:prepare:4, schedule:871, submit:1, rt:9953, total:10830) [2025-01-08 00:50:58.077826] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:39, bw:166251, [delay/us]:prepare:6, schedule:748, submit:3245, rt:6039, total:10040) [2025-01-08 00:50:58.077883] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:11875, [delay/us]:prepare:2, schedule:50, submit:2731, rt:344, total:3128) [2025-01-08 00:50:58.077898] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1002, ref_cnt=3, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:58.078032] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=128] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:7, avg_bw:27704, [delay/us]: prepare:4, schedule:871, submit:1, device:9953, total:10830) [2025-01-08 00:50:58.078049] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:11873, [delay/us]: prepare:2, schedule:50, submit:2731, device:344, total:3128) [2025-01-08 00:50:58.078056] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:39, avg_bw:166224, [delay/us]: prepare:6, schedule:748, submit:3245, device:6039, total:10040) [2025-01-08 00:50:58.078066] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:58.078086] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3958, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:256, total:270) [2025-01-08 00:50:58.078111] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:58.078138] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3958, [delay/us]: prepare:0, schedule:0, submit:600000000, device:256, total:270) [2025-01-08 00:50:58.078155] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:58.078232] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:58.078244] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:58.078252] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:58.078260] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=211kB/s, net_out=212kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:58.078510] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=174] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7454, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:58.078526] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2173474}], scn_array=[cnt:1, {val:1736326257710007003, v:0}], curr_size=122, accum_size=1220, time_cost=15546) [2025-01-08 00:50:58.078548] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:58.078630] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2463, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:58.081529] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:58.087514] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17854, miss_cnt:0, cache_read_size:2183921, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:58.090303] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=96] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:58.093785] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:58.103570] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:50:58.103607] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:50:58.106390] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2183921}], scn_array=[cnt:1, {val:1736326257996509000, v:0}], curr_size=122, accum_size=244, time_cost=12910) [2025-01-08 00:50:58.119656] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84210167}], scn_array=[cnt:1, {val:1736326258104300000, v:0}], curr_size=586, accum_size=1294, time_cost=12967) [2025-01-08 00:50:58.119798] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2150764}], scn_array=[cnt:1, {val:1736326257928067001, v:0}], curr_size=122, accum_size=122, time_cost=15977) [2025-01-08 00:50:58.119918] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=32, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:58.119961] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=143] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1089, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:58.141348] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=83] get number(ret=0, service_type_=0, range=1, base_id=1736326258138656000, start_id=1736326258138656000, end_id=1736326258138656001) [2025-01-08 00:50:58.141756] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:58.141758] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43078208}], scn_array=[cnt:1, {val:1736326258113077000, v:0}], curr_size=589, accum_size=4873, time_cost=24308) [2025-01-08 00:50:58.141919] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=44] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=46, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:58.143305] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58344][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=41] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:50:58.150717] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.159774] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=31] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:58.166708] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3757-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:58.166781] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3757-0-0] [lt=37][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17857, log_task={header:{begin_lsn:{lsn:2173474}, end_lsn:{lsn:2173596}, log_id:17857, min_scn:{val:1736326257710007003, v:0}, max_scn:{val:1736326257710007003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2173352}, prev_proposal_id:1, committed_end_lsn:{lsn:2173474}, data_checksum:940009244, accum_checksum:1413309800, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326258062636, freeze_ts:1736326258062636, submit_ts:1736326258062653, flushed_ts:1736326258081487, gen_to_freeze cost time:0, gen_to_submit cost time:17, submit_to_flush cost time:18834}, fs_cb_begin_ts=1736326258166751, log_life_time=104115) [2025-01-08 00:50:58.176653] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC380-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:58.182866] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2173596}], scn_array=[cnt:1, {val:1736326257710007004, v:0}], curr_size=122, accum_size=122, time_cost=15927) [2025-01-08 00:50:58.189143] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4493-0-0] [lt=38][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17943, log_task={header:{begin_lsn:{lsn:2183921}, end_lsn:{lsn:2184043}, log_id:17943, min_scn:{val:1736326257996509000, v:0}, max_scn:{val:1736326257996509000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2183799}, prev_proposal_id:1, committed_end_lsn:{lsn:2183921}, data_checksum:1715074492, accum_checksum:2944582527, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326258087349, freeze_ts:1736326258087349, submit_ts:1736326258087357, flushed_ts:1736326258106528, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:19171}, fs_cb_begin_ts=1736326258189130, log_life_time=101781) [2025-01-08 00:50:58.196878] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6731165696, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:58.203304] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:50:58.207033] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=119][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2184043}], scn_array=[cnt:1, {val:1736326257996509001, v:0}], curr_size=122, accum_size=366, time_cost=15759) [2025-01-08 00:50:58.211520] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE7F9-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326258210334000, start_id=1736326258210334000, end_id=1736326258210334001) [2025-01-08 00:50:58.213769] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE7F8-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1736326258213023000, start_id=1736326258213023000, end_id=1736326258213023001) [2025-01-08 00:50:58.216943] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=122] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10063, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:58.216971] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18651915}], scn_array=[cnt:1, {val:1736326257928067002, v:0}], curr_size=122, accum_size=244, time_cost=10765) [2025-01-08 00:50:58.217008] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1354, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:58.217034] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2150886}], scn_array=[cnt:1, {val:1736326257928067002, v:0}], curr_size=122, accum_size=244, time_cost=10767) [2025-01-08 00:50:58.217292] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:58.224590] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=17] gc stale ls task succ [2025-01-08 00:50:58.228520] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:58.228550] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:58.233913] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8703A-0-0] [lt=14][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:58.233956] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:58.233967] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:58.239865] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [46106][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=85][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:9240, tid:50418}]) [2025-01-08 00:50:58.250867] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.254586] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=116] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:58.255617] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:50:58.255797] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:50:58.269482] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46503][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCE-0-0] [lt=18][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:50:58.269519] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46503][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCE-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:58.271030] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=16] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:300K/70:970464 disk:300K/70:970464 t1m2:44K/10:557190 t1m2g0:44K/10:557190 t1002m2:192K/46:1139404 t1002m2g0:192K/46:1139404 t1001m2:64K/14:710572 t1001m2g0:64K/14:710572 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:58.279597] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CCE-0-0] [lt=9][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:50:58.282236] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3540) [2025-01-08 00:50:58.282749] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=26] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-01-08 00:50:58.283073] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:50:58.283384] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:58.283483] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-01-08 00:50:58.283498] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:58.283614] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=113] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:58.283630] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193762196900, flush_io_finished_ret_:0, flush_io_finished_round_:1921, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:58.292849] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58244][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4495-0-0] [lt=19][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17944, log_task={header:{begin_lsn:{lsn:2184043}, end_lsn:{lsn:2184165}, log_id:17944, min_scn:{val:1736326257996509001, v:0}, max_scn:{val:1736326257996509001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2183921}, prev_proposal_id:1, committed_end_lsn:{lsn:2184043}, data_checksum:1715074492, accum_checksum:2288821770, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326258189088, freeze_ts:1736326258189088, submit_ts:1736326258189100, flushed_ts:1736326258207088, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:17988}, fs_cb_begin_ts=1736326258292799, log_life_time=103711) [2025-01-08 00:50:58.294251] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=133] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:58.297867] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA0-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:58.297897] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA0-0-0] [lt=26] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:58.298075] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe4a030, session_id:3221580535, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1106, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:58.298118] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221580535, proxy_sessid=0, ret=0) [2025-01-08 00:50:58.298297] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] sock ref clean, do destroy(*s={this:0x2ad8efe4a030, session_id:3221580535, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326257206471, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1106, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:50:58.298357] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] unlock session of tenant(conn.sessid_=3221580535, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:50:58.298372] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221580535, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CCE-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:1001, rec_:1001, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:5, seq_:7}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:37, seq_:45}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:46}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:50:58.298430] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=39] free session successfully(sessid=3221580535) [2025-01-08 00:50:58.298450] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] mark session id unused(sessid=3221580535) [2025-01-08 00:50:58.303854] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=629149696, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:50:58.303909] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=53] === Run print tenant memory usage task === [2025-01-08 00:50:58.303960] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=17] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,088,512 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 631,246,848 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,258,303,674 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 160,801,160 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,038,594,048 max_mem_memstore_can_get_now= 804,005,886 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,258,303,674 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 70,285,300 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 656,101,376 max_mem_memstore_can_get_now= 351,426,548 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,258,303,674 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 240,882,460 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,112,293,376 max_mem_memstore_can_get_now= 1,204,412,368 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:50:58.304272] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=29] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,654,410,240 total_hold= 3,852,468,224 used= 3,467,284,480 freelists_hold= 187,125,760 total_maps= 1,625 total_unmaps= 235 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,088,512 virtual_memory_used= 4,824,498,176 [CHUNK_MGR] 2 MB_CACHE: hold= 83,886,080 count= 40 pushes= 1,809,829 pops= 1,809,789 maps= 1,494 unmaps= 230 [CHUNK_MGR] 4 MB_CACHE: hold= 9,035,776 count= 3 pushes= 6,152 pops= 6,149 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,072 pops= 3,068 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:50:58.304306] INFO print (ob_malloc_time_monitor.cpp:37) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=18] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 30698, delta_count= 1, avg_cost_time= 30698 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-01-08 00:50:58.316309] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185415-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:58.319272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.319314] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.330410] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB56-0-0] [lt=27] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:58.330450] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB56-0-0] [lt=33] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:58.347679] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:58.347741] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=20] current monitor number(seq_=-1) [2025-01-08 00:50:58.351019] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884ED9-0-0] [lt=22] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:58.351479] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.357385] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C296-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:58.357430] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C296-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326257996509002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326257996509002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:58.373496] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58369][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:58.373567] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:50:58.386882] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:50:58.392664] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:58.392700] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-01-08 00:50:58.392711] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:58.392725] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:58.392739] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193762306399, flush_io_finished_ret_:0, flush_io_finished_round_:1917, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:58.397443] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:58.407154] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB64-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193763320987, cache_obj->added_lc()=false, cache_obj->get_object_id()=13575, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:58.409165] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=67, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:58.409413] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:50:58.428640] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8587, offset=2436145}, end_cursor:ObLogCursor{file_id=1, log_id=8588, offset=2436450}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894718050, len:975, log_data_len:305, seq:8587, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:58.429156] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E43-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:58.429268] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E43-0-0] [lt=93] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326257928067004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326257928067004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:58.430224] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=41] ts response task statistics(alloc_count=59598, free_count=29798) [2025-01-08 00:50:58.433564] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E43-0-0] [lt=22] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.719243288040161133e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=385, Throttle Trigger(MB)=231, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.905212950641981706e+02) [2025-01-08 00:50:58.437644] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=9483, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:58.437686] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=8524) [2025-01-08 00:50:58.437765] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=1070, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:58.437857] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=36] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=47, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:58.438589] INFO [LIB] stat (utility.h:1210) [58356][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BD1-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=17, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:58.438626] INFO [LIB] stat (utility.h:1210) [58356][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BD1-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=12080, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:58.438640] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58356][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BD1-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=42, avg_log_gen_to_submit_time=130, avg_log_submit_to_flush_time=10727, avg_log_submit_to_slide_time=11949) [2025-01-08 00:50:58.438736] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=8524, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:58.443335] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=40] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8588, offset=2436450}, end_cursor:ObLogCursor{file_id=1, log_id=8589, offset=2436747}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894720050, len:670, log_data_len:297, seq:8588, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:58.443864] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E43-0-0] [lt=32] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:58.443925] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E43-0-0] [lt=43] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326257928067004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326257928067004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:58.445558] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:58.452097] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.452348] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8589, offset=2436747}, end_cursor:ObLogCursor{file_id=1, log_id=8590, offset=2437044}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894724050, len:373, log_data_len:297, seq:8589, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:58.452761] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=111] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43083198}, readable_log_tail:{lsn:43083198}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:24021570, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:58.452854] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=92] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:58.452887] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:50:58.453650] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8590, offset=2437044}, end_cursor:ObLogCursor{file_id=1, log_id=8591, offset=2437341}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894728050, len:4172, log_data_len:297, seq:8590, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:58.453744] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=26] clear dtl interm result cost(us)(clear_cost=52, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:50:58.453947] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17771, miss_cnt:0, cache_read_size:2173962, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:58.455373] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEBE-0-0] [lt=83] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:58.455471] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEBE-0-0] [lt=91] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:58.459011] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.459583] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.460168] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:50:58.460622] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD86-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326258564117, keepalive_ts:-1}) [2025-01-08 00:50:58.460654] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD86-0-0] [lt=30] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326258564117, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:58.464329] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=656101376, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=417640448, wash_ratio=6, tenant_id=1001) [2025-01-08 00:50:58.464402] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=40] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:50:58.464422] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=17] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2806, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:58.464482] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1112293376, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2108932096, wash_ratio=6, tenant_id=1002) [2025-01-08 00:50:58.464531] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:50:58.464545] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=13] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:58.472821] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] clear dtl interm result cost(us)(clear_cost=4138, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:50:58.474866] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:58.475662] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:58.477980] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2173962}], scn_array=[cnt:1, {val:1736326258210334002, v:0}], curr_size=113, accum_size=479, time_cost=10054) [2025-01-08 00:50:58.478881] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=75] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3540, avg_group_log_size=354) [2025-01-08 00:50:58.478924] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3540) [2025-01-08 00:50:58.478982] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:58.495330] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83F5-0-0] [lt=120] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=7981, avg_log_submit_to_slide_time=49180) [2025-01-08 00:50:58.495404] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:58.511715] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=183] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326258104300001, v:0}, new_max_version={val:1736326258104300001, v:0}, last_min_version={val:1736326257928067000, v:0}, last_max_version={val:1736326257928067000, v:0}, total_time_us=7237, delta_us=407412) [2025-01-08 00:50:58.511756] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326258104300001, v:0}, min_version={val:1736326258104300001, v:0}, max_version={val:1736326258104300001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:58.519304] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=69] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.542230] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.543146] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=11] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:50:58.544122] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=86] PNIO [ratelimit] time: 1736326258544119, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006754, add_bytes: 0 [2025-01-08 00:50:58.544282] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.552175] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.563354] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC375F-0-0] [lt=42] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=24, avg_log_submit_to_flush_time=9292, avg_log_submit_to_slide_time=65478) [2025-01-08 00:50:58.563502] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1455, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:58.576637] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=37] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:58.578819] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:60, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2016942272, recv_hp_rpc_cnt:5127, recv_np_rpc_cnt:103003, recv_lp_rpc_cnt:0, recv_mysql_cnt:9293, recv_task_cnt:3720, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10744299, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=115306 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1827, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:93305, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326257239779}{group_id:10, queue_size:0, recv_req_cnt:11115, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2064, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326228853642}{group_id:2, queue_size:0, recv_req_cnt:29866, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11835, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:4944, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:26534, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=267 pcode=0x1502:cnt=264 pcode=0x717:cnt=190 pcode=0x710:cnt=63 pcode=0x150b:cnt=60, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:50:58.579672] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=852] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:14245829, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8149, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3555150, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=20, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:50:58.580447] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=773] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:13444919, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3541972, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:50:58.581091] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=643] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:22, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:867557071, recv_hp_rpc_cnt:3687, recv_np_rpc_cnt:37449, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:768564, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=64819 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:67031, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:5, nesting_worker_cnt:0, token_change:1736326258372314}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2156, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326218958545}{group_id:2, queue_size:0, recv_req_cnt:15261, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18307, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=336 pcode=0x51c:cnt=258 pcode=0x717:cnt=186 pcode=0x710:cnt=98 pcode=0x150b:cnt=60, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:50:58.581871] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=778] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:56, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9315032787, recv_hp_rpc_cnt:77, recv_np_rpc_cnt:34462, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2906432, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=24767 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:0, recv_req_cnt:6926, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326208268040}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:148346, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736326257592570}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326242277705}{group_id:19, queue_size:0, recv_req_cnt:402, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326204280596}{group_id:2, queue_size:0, recv_req_cnt:36576, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:10940, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=400 pcode=0x1501:cnt=198 pcode=0x1516:cnt=186 pcode=0x717:cnt=181 pcode=0x150b:cnt=120, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:50:58.585577] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD1-0-0] [lt=25] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:58.585621] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD1-0-0] [lt=34] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=48) [2025-01-08 00:50:58.590922] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=48] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:58.593410] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=905] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:58.593465] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58346][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=13] worker thread exit(tenant_->id()=1002, workers_.get_size()=4) [2025-01-08 00:50:58.594029] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:58.597915] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=105] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:58.609831] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=45, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:58.609884] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:58.609906] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:58.609921] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:58.610159] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:50:58.611254] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:58.613514] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9041-0-0] [lt=35] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:58.613539] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9041-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8383, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:58.614089] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9042-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:58.614108] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9042-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8634, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:58.614117] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9042-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=103, avg_log_submit_to_flush_time=5383, avg_log_submit_to_slide_time=8530) [2025-01-08 00:50:58.614397] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:50:58.619073] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=52] BlockGCTimerTask success(ret=0, cost_time_us=17, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:50:58.652345] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.660458] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:58.672394] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EC-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:58.672444] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EC-0-0] [lt=32] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326258210334003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326258210334003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:58.677745] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=56] [COMPACTION DAG STATUS] (start_time=1736326248670429, end_time=1736326258677742, dag_status={}) [2025-01-08 00:50:58.677788] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=40] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:58.677804] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:58.677812] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:58.677820] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:58.677827] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:58.677834] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:58.677842] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:58.677848] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:58.677855] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:58.677862] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677874] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677882] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677890] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677898] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677907] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677914] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677922] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677931] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677939] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677947] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677955] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677963] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677971] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677979] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677987] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.677995] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678003] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678011] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678019] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678027] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678035] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678043] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678051] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678059] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678066] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678074] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678082] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678090] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678098] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678106] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678286] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=178] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678300] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678309] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678318] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678326] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678334] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678343] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678351] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678359] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678367] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678375] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678384] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678392] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678400] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678408] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678416] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678423] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678431] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678439] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678447] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678455] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678463] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678471] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678479] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678487] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678495] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678503] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678511] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678519] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678527] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678535] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678543] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678551] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678559] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678567] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678575] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678583] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678591] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678599] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678607] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678615] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678623] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:58.678633] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:50:58.678641] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:50:58.678649] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:50:58.678654] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:50:58.678659] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:50:58.678664] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:50:58.678670] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:50:58.678675] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:50:58.678680] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:50:58.678685] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:50:58.678692] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:50:58.684615] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5193211286, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:50:58.684792] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=71] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:58.684817] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=23] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5193211286, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:50:58.688764] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A449D-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:58.696676] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=20] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:58.702870] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A8-0-0] [lt=35] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326258701198, keepalive_ts:-1}) [2025-01-08 00:50:58.703315] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A8-0-0] [lt=442] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326258701198, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:58.703352] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58244][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A9-0-0] [lt=49] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326258701563, keepalive_ts:-1}) [2025-01-08 00:50:58.703380] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58244][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A9-0-0] [lt=27] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326258701563, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:58.715862] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6259, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:58.715930] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=375, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:58.719280] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=49] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:50:58.719501] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=24] ====== tenant freeze timer task ====== [2025-01-08 00:50:58.720079] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=9] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:50:58.721627] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=117] get number(ret=0, service_type_=0, range=1, base_id=1736326258720921000, start_id=1736326258720921000, end_id=1736326258720921001) [2025-01-08 00:50:58.722266] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB APPEND COST TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=142, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:50:58.722350] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=82] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=7950, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:50:58.722364] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=13] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:50:58.722371] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=-1138, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:50:58.722378] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=6953, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:50:58.734619] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDBF-0-0] [lt=30][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193763648449, cache_obj->added_lc()=false, cache_obj->get_object_id()=13576, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:58.746826] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326258545466000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=201348, in_cluster_service=true, cluster_version={val:1736326258346557000, v:0}, min_cluster_version={val:1736326258346557000, v:0}, max_cluster_version={val:1736326258346557000, v:0}, get_cluster_version_err=0, cluster_version_delta=400257, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18594, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=604383597, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:58.752446] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.762926] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3763-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:58.762986] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3763-0-0] [lt=30] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:50:58.763035] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3763-0-0] [lt=19][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17863, log_task={header:{begin_lsn:{lsn:2174197}, end_lsn:{lsn:2174319}, log_id:17863, min_scn:{val:1736326258210334004, v:0}, max_scn:{val:1736326258210334004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2174075}, prev_proposal_id:1, committed_end_lsn:{lsn:2174197}, data_checksum:940009244, accum_checksum:3196142213, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326258662943, freeze_ts:1736326258662943, submit_ts:1736326258662956, flushed_ts:1736326258667066, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:4110}, fs_cb_begin_ts=1736326258763011, log_life_time=100068) [2025-01-08 00:50:58.766438] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9046-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=35, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:58.766585] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9046-0-0] [lt=146] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=60463, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:58.768121] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:58.782441] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:58.793103] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83FE-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:58.793139] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC83FE-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=35115, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:58.796147] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D01A-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:58.798483] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=81] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:58.809318] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F0-0-0] [lt=14] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326258806497, keepalive_ts:-1}) [2025-01-08 00:50:58.809359] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F0-0-0] [lt=38] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326258806497, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:58.810539] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:50:58.817282] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326258210334003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=606939, in_cluster_service=true, cluster_version={val:1736326257928067003, v:0}, min_cluster_version={val:1736326257928067003, v:0}, max_cluster_version={val:1736326257928067003, v:0}, get_cluster_version_err=0, cluster_version_delta=889206, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18158, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=486734800, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:58.817729] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58371][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:58.817796] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread created(tenant_->id()=1002, group_id_=1, token=4) [2025-01-08 00:50:58.821958] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=79] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:50:58.822040] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=80] try gc retain ctx [2025-01-08 00:50:58.822050] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-01-08 00:50:58.822083] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326258743968000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326248586728001, v:0}, keep_alive_scn_:{val:1736326253696901001, v:0}, update_ts_:1736326258820793}) [2025-01-08 00:50:58.822891] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269284252000, latest_log_ts:{val:1736326252703023001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:58.823485] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=137] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9387, avg_group_log_size=625) [2025-01-08 00:50:58.823509] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9863) [2025-01-08 00:50:58.823572] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=40] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:44466, max_lsn:84213422, max_scn:{val:1736326258720921000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84213422}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42270382}}, last_submit_lsn:{lsn:84212877}, last_submit_end_lsn:{lsn:84213422}, last_submit_log_id:44466, last_submit_log_pid:1, max_flushed_lsn:{lsn:84212877}, max_flushed_end_lsn:{lsn:84213422}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84213422}, last_slide_log_id:44466, last_slide_scn:{val:1736326258720921000, v:0}, last_slide_lsn:{lsn:84212877}, last_slide_end_lsn:{lsn:84213422}, last_slide_log_pid:1, last_slide_log_accum_checksum:2111681091, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:44467, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84213422}, readable_log_tail:{lsn:84213422}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49996114, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:50:58.823924] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=347] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326258723020, last_flushed_end_lsn:{lsn:84213422}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326258729275, last_flushed_end_lsn:{lsn:84213422}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326258732919, last_flushed_end_lsn:{lsn:84213422}}]) [2025-01-08 00:50:58.823950] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=44467, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326258723020, last_flushed_end_lsn:{lsn:84213422}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326258729275, last_flushed_end_lsn:{lsn:84213422}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326258732919, last_flushed_end_lsn:{lsn:84213422}}]) [2025-01-08 00:50:58.824168] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:58.852601] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.858461] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1002}, {id:1}]) [2025-01-08 00:50:58.863701] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58371][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3765-0-0] [lt=43] init point thread id with(&point=0x5572d50e2a00, idx_=1027, point=[thread id=58371, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58371) [2025-01-08 00:50:58.877965] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] LogLoopThread round_cost_time(us)(round_cost_time=30) [2025-01-08 00:50:58.878345] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1736326258878340, bytes: 868511859, bw: 0.250298 MB/s, add_ts: 1005797, add_bytes: 263978 [2025-01-08 00:50:58.880096] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:50:58.880144] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=44] try gc retain ctx [2025-01-08 00:50:58.880155] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-01-08 00:50:58.880192] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] get_max_applied_scn(scn={val:1736326258720921000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84213422}, palf_committed_end_scn_:{val:1736326258720921000, v:0}, last_check_scn_:{val:1736326258720921000, v:0}, max_applied_cb_scn_:{val:1736326258720921000, v:0}}) [2025-01-08 00:50:58.880325] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=126] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84213422}, readable_log_tail:{lsn:84213422}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49996114, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:58.880446] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=109] finish update min start scn(max_decided_scn={val:1736326258720921000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326248747804001, v:0}, keep_alive_scn_:{val:1736326253770280001, v:0}, update_ts_:1736326258879787}) [2025-01-08 00:50:58.881170] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:58.882459] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4194, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:58.882489] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4076) [2025-01-08 00:50:58.882527] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=249, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:58.882796] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=35, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:58.884321] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6795-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:58.884356] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6795-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6600, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:58.884369] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6795-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=15, avg_log_gen_to_submit_time=93, avg_log_submit_to_flush_time=4591, avg_log_submit_to_slide_time=6507) [2025-01-08 00:50:58.884474] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4076, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:58.889284] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58371][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44A1-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:58.889333] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58371][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44A1-0-0] [lt=46] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:58.890531] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:58.896301] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=21] ====== check clog disk timer task ====== [2025-01-08 00:50:58.896383] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=78] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:50:58.896748] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.897378] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.897461] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.897490] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:58.919506] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:58.923845] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30] LogLoopThread round_cost_time(us)(round_cost_time=429) [2025-01-08 00:50:58.930245] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=51] ====== check clog disk timer task ====== [2025-01-08 00:50:58.930368] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=117] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:58.931257] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=40] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:58.931526] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=497, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:58.950603] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9387) [2025-01-08 00:50:58.952742] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:58.968921] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=31] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:50:58.975062] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=54] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:58.975559] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C8A-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:58.975892] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-01-08 00:50:58.976964] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C8A-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:58.994500] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3852, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:58.994573] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=67] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1626, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:58.994643] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58213][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB5D-0-0] [lt=31] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:58.994677] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58213][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB5D-0-0] [lt=26] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:58.994808] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=48, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:58.998967] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:59.010820] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=92] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:50:59.012238] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] ====== checkpoint timer task ====== [2025-01-08 00:50:59.012861] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:59.013109] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=36] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:59.013172] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=62] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:59.013609] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=145] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269284252000, latest_log_ts:{val:1736326252703023001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:59.013825] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=261] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:50:59.013863] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=30] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:59.013877] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326252703023001, v:0}) [2025-01-08 00:50:59.013885] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:50:59.013891] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:50:59.013903] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:59.013943] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=34] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:59.013961] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:59.014024] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:59.014071] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=45] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:59.014084] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:50:59.014796] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=58] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:50:59.016045] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8405-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:59.017461] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=36] get number(ret=0, service_type_=0, range=1, base_id=1736326259016684000, start_id=1736326259016684000, end_id=1736326259016684001) [2025-01-08 00:50:59.017873] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:59.018057] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:59.025555] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D32-0-0] [lt=17] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:59.052828] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.060186] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B91-0-0] [lt=87] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:59.077016] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:59.080802] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=158] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=68, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:59.090311] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:59.090394] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:52575, [delay/us]:prepare:8, schedule:607, submit:45, rt:1961, total:2622) [2025-01-08 00:50:59.090421] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:59.090502] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=63] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52573, [delay/us]: prepare:8, schedule:607, submit:45, device:1961, total:2622) [2025-01-08 00:50:59.090518] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:59.090558] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:72794, [delay/us]:prepare:16, schedule:322, submit:250, rt:2664, total:3254) [2025-01-08 00:50:59.090595] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:59.090637] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:72791, [delay/us]: prepare:16, schedule:322, submit:250, device:2664, total:3254) [2025-01-08 00:50:59.090655] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:59.090743] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=70] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20218, [delay/us]:prepare:6, schedule:16, submit:0, rt:1205, total:1228) [2025-01-08 00:50:59.090772] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:39, bw:161747, [delay/us]:prepare:6, schedule:733, submit:249, rt:2869, total:3858) [2025-01-08 00:50:59.090811] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:59.090848] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20220, [delay/us]: prepare:6, schedule:16, submit:0, device:1205, total:1228) [2025-01-08 00:50:59.090864] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:161768, [delay/us]: prepare:6, schedule:733, submit:249, device:2869, total:3858) [2025-01-08 00:50:59.090874] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:59.090894] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4044, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:868, total:878) [2025-01-08 00:50:59.090917] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:59.091014] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=89] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4043, [delay/us]: prepare:0, schedule:0, submit:600000000, device:868, total:878) [2025-01-08 00:50:59.091035] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:59.091176] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=130] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:59.091188] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:59.091239] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:59.091251] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=265kB/s, net_out=263kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:59.091667] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:59.091978] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=70] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17864, miss_cnt:0, cache_read_size:2185141, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:59.092073] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=345, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:59.092093] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:59.093749] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=7] success to update min reserved snapshot(reserved_snapshot=1736324454278994000, duration=1800, snapshot_gc_ts=1736326254278994000) [2025-01-08 00:50:59.094390] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:59.096904] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=42] update reserved snapshot finished(cost_ts=3157, reserved_snapshot=1736324454278994000) [2025-01-08 00:50:59.112879] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:59.112968] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:59.119443] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58213][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC5-0-0] [lt=27] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:59.119485] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58213][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC5-0-0] [lt=34] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:59.149014] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=74] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=102, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:59.152998] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.160906] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=30] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:59.160957] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=59] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:59.164437] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC383-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:59.165985] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=106] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4395, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:59.166014] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1333) [2025-01-08 00:50:59.166049] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1231, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:59.166381] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=70, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:59.168975] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58371][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6B6-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:50:59.181107] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=55] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:50:59.183057] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326259179773000, start_id=1736326259179773000, end_id=1736326259179773001) [2025-01-08 00:50:59.184454] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18657231, this={dio_aligned_buf:{buf_write_offset:496, buf_padding_size:3600, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:32118, truncate_used_ts:6670}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=8192, aligned_block_offset=18653184, buf_len=545, total_write_size=15529, total_write_size_after_dio=430080, ob_pwrite_used_ts=396671, count=101, each_pwrite_cost=3927) [2025-01-08 00:50:59.188178] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=154] ====== check clog disk timer task ====== [2025-01-08 00:50:59.188284] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=98] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:50:59.199572] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:59.201440] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=53] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=503312, clean_num=31457) [2025-01-08 00:50:59.211029] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:50:59.213044] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=169] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:59.215439] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=30] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:59.215534] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326254278994000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:59.215748] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=211] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326254278994000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326254278994000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:59.217967] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=25, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:59.218231] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:59.224699] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:50:59.228816] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:59.228846] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:59.231909] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87040-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:59.231953] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:59.231964] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:59.253157] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.261817] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:59.261843] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:59.261961] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E4-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:59.262013] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E4-0-0] [lt=48] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:59.274614] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=195] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:59.275992] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=39] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:59.276083] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326255686329000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:59.276110] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326255686329000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326255686329000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:59.282711] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4076) [2025-01-08 00:50:59.283279] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:50:59.294450] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:59.298331] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=14] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:320K/74:221168 disk:320K/74:221168 t1m2:52K/12:359149 t1m2g0:52K/12:359149 t1002m2:192K/45:163034 t1002m2g0:192K/45:163034 t1001m2:76K/17:276134 t1001m2g0:76K/17:276134 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:59.299812] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=9] finish to resize log external storage handler(current_ts=1736326259299806, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:50:59.299844] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=30] snapshot_traffic_info once [2025-01-08 00:50:59.300047] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA1-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:59.300079] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA1-0-0] [lt=29] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:59.318624] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185416-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:59.321520] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.321576] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.323418] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58356][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF27-0-0] [lt=27] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326259318569, keepalive_ts:-1}) [2025-01-08 00:50:59.323465] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58356][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF27-0-0] [lt=45] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326259318569, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:59.323872] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=35] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:50:59.323903] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=27] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:50:59.323915] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:50:59.323926] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:59.327802] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:50:59.329028] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=1222] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:59.329400] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2152228}], scn_array=[cnt:1, {val:1736326259179773001, v:0}], curr_size=122, accum_size=366, time_cost=14611) [2025-01-08 00:50:59.329673] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3392, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:59.329695] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18653680}], scn_array=[cnt:1, {val:1736326259179773001, v:0}], curr_size=122, accum_size=789, time_cost=14977) [2025-01-08 00:50:59.329738] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=544, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:59.330076] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:59.330120] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58372][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:59.331366] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:50:59.331640] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58373][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:59.331712] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58373][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3877-0-0] [lt=43] init point thread id with(&point=0x5572d50e2a80, idx_=1029, point=[thread id=58373, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58373) [2025-01-08 00:50:59.336524] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58374][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:59.336635] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:50:59.349814] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:59.353245] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=114][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.367835] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:59.369570] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=36] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:59.369667] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326255577817000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:59.370072] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1959C616-0-0] [lt=58] get number(ret=0, service_type_=0, range=1, base_id=1736326259368658000, start_id=1736326259368658000, end_id=1736326259368658001) [2025-01-08 00:50:59.393542] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:59.393637] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=87] temporary file module use no blocks [2025-01-08 00:50:59.393648] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:59.393663] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:59.393677] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193763306612, flush_io_finished_ret_:0, flush_io_finished_round_:1918, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:59.397969] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C297-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:59.398012] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C297-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326259001815002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326259001815002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:59.399010] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=17] gc stale ls task succ [2025-01-08 00:50:59.400910] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=40] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:59.400936] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-01-08 00:50:59.400946] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:59.400960] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:59.400974] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193763313917, flush_io_finished_ret_:0, flush_io_finished_round_:3746, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:59.402020] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6726971392, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:59.402307] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] ====== tenant freeze timer task ====== [2025-01-08 00:50:59.402331] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=15] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70075580, tenant_id=1001) [2025-01-08 00:50:59.402356] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:59.406401] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58375][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:59.406540] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] worker thread created(tenant_->id()=1002, group_id_=1, token=5) [2025-01-08 00:50:59.410698] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22] ====== tenant freeze timer task ====== [2025-01-08 00:50:59.413168] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:50:59.414811] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=80, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:59.424086] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=112] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:59.424604] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58375][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8412-0-0] [lt=112] init point thread id with(&point=0x5572d50e2b00, idx_=1031, point=[thread id=58375, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58375) [2025-01-08 00:50:59.432148] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17] get number(ret=0, service_type_=0, range=1, base_id=1736326259427727000, start_id=1736326259427727000, end_id=1736326259427727001) [2025-01-08 00:50:59.437110] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1A79C297-0-0] [lt=40] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.671243333816528320e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=383, Throttle Trigger(MB)=230, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=8.103636458226476407e+02) [2025-01-08 00:50:59.437585] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=78] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:59.439827] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46506][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB69-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193764353659, cache_obj->added_lc()=false, cache_obj->get_object_id()=13577, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:59.445497] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=48] success to update min reserved snapshot(reserved_snapshot=1736324455577817000, duration=1800, snapshot_gc_ts=1736326255577817000) [2025-01-08 00:50:59.453304] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.459321] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.461645] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.464631] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] update reserved snapshot finished(cost_ts=19137, reserved_snapshot=1736324455577817000) [2025-01-08 00:50:59.467886] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=27] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8591, offset=2437341}, end_cursor:ObLogCursor{file_id=1, log_id=8592, offset=2437646}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89472c050, len:3875, log_data_len:305, seq:8591, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:59.468259] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E44-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:59.468321] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E44-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326259179773002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326259179773002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:59.471969] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46386][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BEC4-0-0] [lt=80] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:50:59.475080] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17782, miss_cnt:0, cache_read_size:2175295, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:59.475157] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=56] succ to submit log task to replay service(to_submit_lsn={lsn:2175229}, to_submit_scn={val:1736326259205543002, v:0}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326259471401, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2175295}, next_to_submit_scn_:{val:1736326259205543003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2175173}, end_lsn:{lsn:2175295}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2175295}, readable_log_tail:{lsn:2175295}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64929473, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326259205543002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326259205543002, v:0}, data_checksum:2124447694, flag:38384}}, init_mode_version:0, accumulate_checksum:430356983, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326259205543002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17782, miss_cnt:0, cache_read_size:2175295, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:59.475492] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50613][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C297-0-0] [lt=100] get number(ret=0, service_type_=0, range=1, base_id=1736326259474081000, start_id=1736326259474081000, end_id=1736326259474081001) [2025-01-08 00:50:59.476655] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:59.479078] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=1960, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:59.479220] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=139] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=8910) [2025-01-08 00:50:59.479247] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=322, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:59.479303] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=51] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=15, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:59.479319] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=31] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:59.482864] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:491) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E44-0-0] [lt=8] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=0, MAX_GAP=900000000000, new_sync_scn={val:1736326257928067004, v:0}, new_replayable_scn={val:1736326257928067004, v:0}, new_readable_scn={val:1736326257928067004, v:0}, old_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326256925355002, replayable_scn:1736326256925355002, readable_scn:1736326256925355002, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}) [2025-01-08 00:50:59.483480] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4076, avg_group_log_size=370) [2025-01-08 00:50:59.483498] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4076) [2025-01-08 00:50:59.483511] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:59.484192] INFO [LIB] stat (utility.h:1210) [58299][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BE1-0-0] [lt=124] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=15, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:59.484371] INFO [LIB] stat (utility.h:1210) [58299][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BE1-0-0] [lt=176] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4823, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:59.484464] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58299][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BE1-0-0] [lt=91] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=113, avg_log_submit_to_flush_time=2542, avg_log_submit_to_slide_time=4709) [2025-01-08 00:50:59.484952] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=77] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=8910, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:59.490522] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=50] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8592, offset=2437646}, end_cursor:ObLogCursor{file_id=1, log_id=8593, offset=2437943}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894730050, len:3570, log_data_len:297, seq:8592, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:59.490898] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E44-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:59.491008] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E44-0-0] [lt=23] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326259179773002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326259179773002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:59.500740] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58375][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44AD-0-0] [lt=50] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=51, avg_log_submit_to_flush_time=1893, avg_log_submit_to_slide_time=24752) [2025-01-08 00:50:59.500782] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58375][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44AD-0-0] [lt=40][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17956, log_task={header:{begin_lsn:{lsn:2185507}, end_lsn:{lsn:2185629}, log_id:17956, min_scn:{val:1736326259001815003, v:0}, max_scn:{val:1736326259001815003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2185385}, prev_proposal_id:1, committed_end_lsn:{lsn:2185507}, data_checksum:1715074492, accum_checksum:1044503145, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326259398163, freeze_ts:1736326259398163, submit_ts:1736326259398260, flushed_ts:1736326259405368, gen_to_freeze cost time:0, gen_to_submit cost time:97, submit_to_flush cost time:7108}, fs_cb_begin_ts=1736326259500712, log_life_time=102549) [2025-01-08 00:50:59.500844] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=148] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:59.511748] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8593, offset=2437943}, end_cursor:ObLogCursor{file_id=1, log_id=8594, offset=2438240}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894734050, len:3273, log_data_len:297, seq:8593, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:59.511978] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:59.514639] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=41] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8594, offset=2438240}, end_cursor:ObLogCursor{file_id=1, log_id=8595, offset=2438537}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894738050, len:2976, log_data_len:297, seq:8594, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:59.519239] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.527132] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326259001815001, v:0}, new_max_version={val:1736326259001815001, v:0}, last_min_version={val:1736326258724473003, v:0}, last_max_version={val:1736326258724473003, v:0}, total_time_us=6850, delta_us=525313) [2025-01-08 00:50:59.527217] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=41] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326259001815001, v:0}, min_version={val:1736326259001815001, v:0}, max_version={val:1736326259001815001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:59.542160] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.544239] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.551517] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1736326259551514, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007395, add_bytes: 0 [2025-01-08 00:50:59.553420] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.566473] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58375][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6BE-0-0] [lt=63] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=2968, avg_log_submit_to_slide_time=21483) [2025-01-08 00:50:59.566558] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:59.577511] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=85] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:59.591896] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:59.594621] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:59.602620] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:59.613518] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=80] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:59.615302] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:59.615397] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:50:59.615418] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1332) [2025-01-08 00:50:59.615433] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:59.617350] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=58] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:50:59.619911] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC841A-0-0] [lt=41] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=97, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:59.619948] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC841A-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6989, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:59.621592] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8419-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:59.621613] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8419-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6480, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:59.621623] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58375][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8419-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=4609, avg_log_submit_to_slide_time=6370) [2025-01-08 00:50:59.653602] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=107][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.661461] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=110] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:59.662608] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B93-0-0] [lt=31] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:59.685622] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=21] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5195090211, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:59.685726] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:59.685742] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a1eb0) [2025-01-08 00:50:59.685752] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5195090211, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:59.699371] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44B1-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:59.706396] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] no need to update min/max version(new_version={val:1736326259305391000, v:0}, min_version={val:1736326259305391000, v:0}, max_version={val:1736326259305391000, v:0}) [2025-01-08 00:50:59.706935] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5ED-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:59.707003] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5ED-0-0] [lt=51] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326259205543004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326259205543004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:59.717377] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3429, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:59.717433] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=571, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:59.723526] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AC6C-0-0] [lt=20] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=113) [2025-01-08 00:50:59.731504] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=26] [APPLY STAT CB APPEND COST TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=138, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:50:59.731533] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=29] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=7548, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:50:59.731543] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:50:59.731550] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=-1898, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:50:59.731557] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=5789, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:50:59.734659] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:50:59.737540] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5ED-0-0] [lt=155] get number(ret=0, service_type_=0, range=1, base_id=1736326259736581000, start_id=1736326259736581000, end_id=1736326259736581001) [2025-01-08 00:50:59.742440] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:59.754142] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.763417] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3777-0-0] [lt=60] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:59.763548] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58375][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3777-0-0] [lt=108] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:59.768347] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=112] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:59.782720] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=48] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:59.789300] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=137] [APPLY STAT CB APPEND COST TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=150, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:50:59.789377] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=76] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=6249, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:50:59.789391] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:50:59.789398] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=-807, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:50:59.789406] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=5592, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:50:59.799846] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50625][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D01D-0-0] [lt=213][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:59.803068] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:59.811293] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6C3-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:59.811371] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6C3-0-0] [lt=76] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23862, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:59.815230] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=91] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1132416, replace_num=15728) [2025-01-08 00:50:59.815261] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:59.824514] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=79] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:59.835628] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A71-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326259835599}) [2025-01-08 00:50:59.835743] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285964-0-0] [lt=24] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326259835718, sql_renew_time:1736326259835718, is_mark_fail:false}]) [2025-01-08 00:50:59.835784] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285964-0-0] [lt=39] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326259835599}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326259835718, sql_renew_time:1736326259835718, is_mark_fail:false}]) [2025-01-08 00:50:59.853091] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=36] ls blacklist refresh finish(cost_time=19249) [2025-01-08 00:50:59.855633] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=139][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.858792] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=57] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1002}, {id:1}, {id:1001}]) [2025-01-08 00:50:59.859525] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=20] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:59.866007] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6C4-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:59.866104] INFO [LIB] stat (utility.h:1210) [58375][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6C4-0-0] [lt=94] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12338, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:59.885984] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1736326259885980, bytes: 868877277, bw: 0.345847 MB/s, add_ts: 1007640, add_bytes: 365418 [2025-01-08 00:50:59.892859] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=57] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:50:59.893536] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=27] ====== checkpoint timer task ====== [2025-01-08 00:50:59.895019] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=21] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:59.895068] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=49] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:59.895170] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=25] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:59.895377] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=202] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:50:59.895436] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=51] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:59.895450] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1736326252704413001, v:0}) [2025-01-08 00:50:59.895459] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:50:59.895467] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:50:59.895481] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:59.895543] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:59.895563] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:59.895583] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:59.895694] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=108] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:59.895706] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:50:59.896982] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=43] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.897304] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.897338] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.897362] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=63] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:59.910880] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58375][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44B5-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:59.910944] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58375][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44B5-0-0] [lt=60] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:59.912255] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:59.918997] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=71] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1755, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:59.923708] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=10520, avg_group_log_size=618) [2025-01-08 00:50:59.923794] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=80] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10044) [2025-01-08 00:50:59.932865] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326259932830], gts:1736326259932392000, latest_srr:[mts=1736326259932830]}) [2025-01-08 00:50:59.932898] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:59.932922] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326259932911], gts:1736326259932392000, latest_srr:[mts=1736326259932911]}) [2025-01-08 00:50:59.933539] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326259933442], gts:1736326259932392000, latest_srr:[mts=1736326259933442]}) [2025-01-08 00:50:59.951327] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=60] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:50:59.953497] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=42] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12813840384, fast_delete_log_mode=false) [2025-01-08 00:50:59.955315] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9931) [2025-01-08 00:50:59.955746] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:59.976856] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:59.983437] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=62] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:59.986722] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C8B-0-0] [lt=54][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:59.988088] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C8B-0-0] [lt=57][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:59.989925] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1761, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:59.989958] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3661) [2025-01-08 00:50:59.989980] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=156, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:59.990021] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:59.991281] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67A0-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:59.991317] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67A0-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3546, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:59.991330] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67A0-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=115, avg_log_submit_to_flush_time=2008, avg_log_submit_to_slide_time=3430) [2025-01-08 00:50:59.991406] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3661, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:00.005590] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:00.018449] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:00.018672] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:00.019388] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=38] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:00.021330] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2056, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:00.021362] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=107, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:00.021523] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=76, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:00.021591] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1755) [2025-01-08 00:51:00.021786] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1755) [2025-01-08 00:51:00.027896] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=100] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:00.028348] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:51:00.055865] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=97][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.077692] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=52] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:00.087487] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB6C-0-0] [lt=64][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193765001317, cache_obj->added_lc()=false, cache_obj->get_object_id()=13578, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:00.093339] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:00.096993] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=8] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:00.097094] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=90] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:00.097112] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=14] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:00.097134] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=21] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:00.101333] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=52] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:51:00.101512] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:00.101567] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:10, bw:40507, [delay/us]:prepare:8, schedule:142, submit:66, rt:1581, total:1798) [2025-01-08 00:51:00.101584] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:00.101642] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:10, avg_bw:40508, [delay/us]: prepare:8, schedule:142, submit:66, device:1581, total:1798) [2025-01-08 00:51:00.101656] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:00.101697] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:15, bw:68864, [delay/us]:prepare:12, schedule:289, submit:263, rt:1208, total:1774) [2025-01-08 00:51:00.101708] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:00.101743] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:15, avg_bw:68867, [delay/us]: prepare:12, schedule:289, submit:263, device:1208, total:1774) [2025-01-08 00:51:00.101755] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:00.101777] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16205, [delay/us]:prepare:7, schedule:706, submit:1, rt:3362, total:4077) [2025-01-08 00:51:00.101799] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4291, iops:42, bw:178256, [delay/us]:prepare:13, schedule:150, submit:248, rt:1974, total:2387) [2025-01-08 00:51:00.101809] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:00.101837] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16205, [delay/us]: prepare:7, schedule:706, submit:1, device:3362, total:4077) [2025-01-08 00:51:00.101853] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4291, avg_iops:42, avg_bw:178265, [delay/us]: prepare:13, schedule:150, submit:248, device:1974, total:2387) [2025-01-08 00:51:00.101863] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:00.101883] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4051, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:366, total:378) [2025-01-08 00:51:00.101907] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:00.101984] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=72] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4051, [delay/us]: prepare:0, schedule:0, submit:600000000, device:366, total:378) [2025-01-08 00:51:00.102004] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:00.102021] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:00.102030] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:00.102038] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:00.102047] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=336kB/s, net_out=254kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:00.102339] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=30] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:00.104633] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=23] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:00.105513] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB71-0-0] [lt=50] get number(ret=0, service_type_=0, range=1, base_id=1736326260105290000, start_id=1736326260105290000, end_id=1736326260105290001) [2025-01-08 00:51:00.105818] INFO [LIB] stat (utility.h:1210) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB71-0-0] [lt=27] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=87, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:00.107112] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=216] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:00.107158] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=44] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:00.109568] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084876-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:00.109779] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084876-0-0] [lt=158] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:00.109798] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084876-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:00.112961] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:00.114271] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=12] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:00.116240] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=40] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:00.118059] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=38] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:00.118099] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:00.120089] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084876-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:00.120180] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084876-0-0] [lt=38] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:00.120261] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084876-0-0] [lt=78] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:00.121648] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1755, avg_group_log_size=159) [2025-01-08 00:51:00.121767] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=113] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1755) [2025-01-08 00:51:00.128597] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=12] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:00.131686] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=46] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:00.133983] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=45] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:00.134024] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:00.134912] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17875, miss_cnt:0, cache_read_size:2186474, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:00.135017] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=101, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:00.135037] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:00.135134] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=101][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18654768}], scn_array=[cnt:1, {val:1736326259932392001, v:0}], curr_size=122, accum_size=122, time_cost=10510) [2025-01-08 00:51:00.136517] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084876-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:00.136606] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084876-0-0] [lt=39] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:00.136624] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084876-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:00.136677] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=13] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:00.137748] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=1066] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:00.137804] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=46] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:00.137816] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=11] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:00.137838] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084876-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:00.137877] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084876-0-0] [lt=38] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:00.137907] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=10] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:00.137917] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:00.137937] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084876-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:00.137949] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084876-0-0] [lt=12] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:00.137959] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=4] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:00.137964] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=5] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:00.137987] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084876-0-0] [lt=21] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:00.138000] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084876-0-0] [lt=12] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:00.140412] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:00.140464] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=47] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:00.140477] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=12] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:00.140510] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:00.140522] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:00.140529] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:00.140550] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=17] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:00.140588] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084876-0-0] [lt=37] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:00.153881] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=80, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:00.156096] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.161823] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:00.165019] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC386-0-0] [lt=61][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:00.165702] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:00.180418] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=81] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326256134366, last_first_register_time_us=1736324449183670, register_time_us=1736326256134366, curr_time_us=1736326260180414) [2025-01-08 00:51:00.180555] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=53] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326260180472, last_submit_register_req_time_us=1736326260180472, register_parent_reason=1) [2025-01-08 00:51:00.180578] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:00.206422] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=298] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:00.218857] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:00.221559] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58372][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67A3-0-0] [lt=235] init point thread id with(&point=0x5572d50e2a40, idx_=1028, point=[thread id=58372, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58372) [2025-01-08 00:51:00.221772] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:1736326259932392001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2153438}, palf_committed_end_scn_:{val:1736326259932392001, v:0}, last_check_scn_:{val:1736326259932392001, v:0}, max_applied_cb_scn_:{val:1736326259932392001, v:0}}) [2025-01-08 00:51:00.221806] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2153438}, readable_log_tail:{lsn:2153438}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64951330, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:00.223330] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=16, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:00.224864] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=97] gc stale ls task succ [2025-01-08 00:51:00.227454] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:00.230815] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:00.230865] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:00.237798] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87046-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:00.237878] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:00.237892] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:00.250515] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=30] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:00.250552] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=32] start lib cache evict(tenant_id=1, mem_hold=93765632, mem_limit=107374180, cache_obj_num=306, cache_node_num=306) [2025-01-08 00:51:00.250585] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=31] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=93765632, mem_limit=107374180, cache_obj_num=306, cache_node_num=306) [2025-01-08 00:51:00.250597] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:00.253474] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=306, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}]) [2025-01-08 00:51:00.254419] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=942] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:00.256209] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=125][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.263941] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58375][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3781-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:00.266102] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1439, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:00.266136] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=90, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:00.266255] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=70] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=76, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:00.266317] INFO pn_print_diag_info (group.c:755) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=40] PNIO client:0x2ad88e1fd4a8_"192.168.159.137:2882"_"192.168.159.137:2882"_147_1736322505625440_1, write_queue=0/0, write=16477/15405826, read=16477/4060016, doing=0, done=16477, write_time=17885866, read_time=1814, process_time=2872 [2025-01-08 00:51:00.266338] INFO pn_print_diag_info (group.c:755) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO client:0x2ad88e0aaa58_"192.168.159.139:2882"_"192.168.159.139:2882"_128_1736322504703688_1, write_queue=1/279, write=69007/67305933, read=69007/15399230, doing=1, done=69007, write_time=84371762, read_time=36073, process_time=41693 [2025-01-08 00:51:00.266347] INFO pn_print_diag_info (group.c:755) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x2ad88e0874e8_"192.168.159.138:2882"_"192.168.159.138:2882"_126_1736322504683741_1, write_queue=0/0, write=69586/57097609, read=69586/15409679, doing=0, done=69587, write_time=85354652, read_time=14085, process_time=19176 [2025-01-08 00:51:00.266355] INFO pn_print_diag_info (group.c:768) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2ad8ae186048_"192.168.159.139:45082"_151_1736322506111955, write_queue=0/0, write=102452/61159699, read=102452/77335619, doing=0, done=102452, write_time=119838186, read_time=916247, process_time=924516 [2025-01-08 00:51:00.266392] INFO pn_print_diag_info (group.c:768) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=35] PNIO server:0x2ad88e1ff4a8_"192.168.159.138:42022"_150_1736322505867001, write_queue=0/0, write=105015/66900272, read=105015/82072746, doing=0, done=105015, write_time=123293785, read_time=697079, process_time=705019 [2025-01-08 00:51:00.266399] INFO pn_print_diag_info (group.c:768) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO server:0x2ad88e1fea98_"192.168.159.137:41292"_149_1736322505627594, write_queue=0/0, write=16477/4060016, read=16477/15405826, doing=0, done=16477, write_time=17803517, read_time=3161, process_time=4684 [2025-01-08 00:51:00.266405] INFO pn_print_diag_info (group.c:775) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client_send:155072/139809986, client_queue_time=160685593, cnt=3, server_send:223944/132119987, server_queue_time=131885663, cnt=3 [2025-01-08 00:51:00.279822] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58375][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF44-0-0] [lt=29] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326260382089, keepalive_ts:-1}) [2025-01-08 00:51:00.279857] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58375][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF44-0-0] [lt=33] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326260382089, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:00.283399] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:51:00.284249] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:00.284301] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=45] temporary file module use no blocks [2025-01-08 00:51:00.284315] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:00.284329] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:00.284342] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193764197938, flush_io_finished_ret_:0, flush_io_finished_round_:1923, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:00.294689] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=11] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:00.302330] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA2-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:00.302391] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA2-0-0] [lt=56] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:00.304086] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:00.304301] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:00.309658] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58375][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44BD-0-0] [lt=15][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17964, log_task={header:{begin_lsn:{lsn:2186474}, end_lsn:{lsn:2186596}, log_id:17964, min_scn:{val:1736326260017888001, v:0}, max_scn:{val:1736326260017888001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2186361}, prev_proposal_id:1, committed_end_lsn:{lsn:2186474}, data_checksum:1715074492, accum_checksum:3293274845, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326260207607, freeze_ts:1736326260207607, submit_ts:1736326260207624, flushed_ts:1736326260212563, gen_to_freeze cost time:0, gen_to_submit cost time:17, submit_to_flush cost time:4939}, fs_cb_begin_ts=1736326260309642, log_life_time=102035) [2025-01-08 00:51:00.310049] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=190] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:296K/70:269314 disk:296K/70:269314 t1m2:44K/10:548192 t1m2g0:44K/10:548192 t1002m2:184K/46:196974 t1002m2g0:184K/46:196974 t1001m2:68K/14:307804 t1001m2g0:68K/14:307804 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:00.321444] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185417-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:00.321829] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.321922] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.325957] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:00.327106] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=5] [APPLY STAT CB APPEND COST TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=105, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:00.327290] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=83] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=12111, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:00.327313] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=22] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:00.327353] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=38] [APPLY STAT CB EXECUTE TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=-1841, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:00.327366] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=12] [APPLY STAT CB TOTAL TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=10374, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:00.327471] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58372][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=42] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:51:00.336300] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:00.336789] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58369][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=136] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:51:00.348157] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=68] current monitor number(seq_=-1) [2025-01-08 00:51:00.354185] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:00.356389] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:00.356439] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.359474] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C617-0-0] [lt=17][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:51:00.375507] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=63] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2176449}, scn={val:1736326260213059002, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:00.375499] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:22, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326260374989, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2176393}, next_to_submit_scn_:{val:1736326260213059002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:56, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2176393}, end_lsn:{lsn:2176505}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2176505}, readable_log_tail:{lsn:2176505}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64928263, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326260213059002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326260213059002, v:0}, data_checksum:3347137827, flag:59305}}, init_mode_version:0, accumulate_checksum:1785951613, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326260213059002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17792, miss_cnt:0, cache_read_size:2176505, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2176449}, scn_:{val:1736326260213059002, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326260369746, first_handle_ts_:1736326260375495, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad966804150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:51:00.375810] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=39] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324455577817000) [2025-01-08 00:51:00.375854] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2176449}, scn_:{val:1736326260213059002, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326260369746, first_handle_ts_:1736326260375793, replay_cost_:58, retry_cost_:-1, read_log_buf_:0x2ad966804160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326260374989, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2176505}, next_to_submit_scn_:{val:1736326260213059003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2176393}, end_lsn:{lsn:2176505}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2176505}, readable_log_tail:{lsn:2176505}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64928263, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326260213059002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326260213059002, v:0}, data_checksum:3347137827, flag:59305}}, init_mode_version:0, accumulate_checksum:1785951613, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326260213059002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17792, miss_cnt:0, cache_read_size:2176505, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=18, ret=0) [2025-01-08 00:51:00.390274] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4104) [2025-01-08 00:51:00.392305] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=59] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:00.392344] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=25] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:00.392360] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=88) [2025-01-08 00:51:00.393886] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:00.393919] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-01-08 00:51:00.393981] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=59] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:00.393998] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:00.394011] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193764307551, flush_io_finished_ret_:0, flush_io_finished_round_:1919, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:00.401267] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:00.401354] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=80] temporary file module use no blocks [2025-01-08 00:51:00.401368] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:00.401380] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:00.401392] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193764314846, flush_io_finished_ret_:0, flush_io_finished_round_:3747, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:00.404954] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=90] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:00.404994] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:00.405009] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:00.405017] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:00.405024] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:00.405032] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=95) [2025-01-08 00:51:00.406935] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=15] gc stale ls task succ [2025-01-08 00:51:00.407271] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:00.408504] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:00.408688] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58375][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=118] worker thread exit(tenant_->id()=1002, workers_.get_size()=5) [2025-01-08 00:51:00.416246] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1736326260415603000, start_id=1736326260415603000, end_id=1736326260415603001) [2025-01-08 00:51:00.419281] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=102] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:00.422509] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=97, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:00.424928] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=101] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3505, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:00.424984] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=519, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:00.425016] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:00.434024] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=562, get_gts_with_stc_cnt=39, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=36, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:00.434071] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=34] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=505, get_gts_with_stc_cnt=71, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=71, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:00.456598] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.459092] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.459504] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.471324] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46493][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB72-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193765385156, cache_obj->added_lc()=false, cache_obj->get_object_id()=13579, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:00.477075] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:00.483673] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:00.494826] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D22484648-0-0] [lt=46] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:51:00.515070] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44C1-0-0] [lt=23] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=30, avg_log_submit_to_flush_time=2108, avg_log_submit_to_slide_time=25896) [2025-01-08 00:51:00.515130] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44C1-0-0] [lt=58][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17966, log_task={header:{begin_lsn:{lsn:2186718}, end_lsn:{lsn:2186840}, log_id:17966, min_scn:{val:1736326260017888003, v:0}, max_scn:{val:1736326260017888003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2186596}, prev_proposal_id:1, committed_end_lsn:{lsn:2186718}, data_checksum:1715074492, accum_checksum:3693652811, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326260410352, freeze_ts:1736326260410352, submit_ts:1736326260410417, flushed_ts:1736326260412046, gen_to_freeze cost time:0, gen_to_submit cost time:65, submit_to_flush cost time:1629}, fs_cb_begin_ts=1736326260515035, log_life_time=104683) [2025-01-08 00:51:00.516630] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=116] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:00.519213] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=36] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.524873] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AC78-0-0] [lt=140] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.719243288040161133e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=385, Throttle Trigger(MB)=231, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.905212950641981706e+02) [2025-01-08 00:51:00.525887] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C298-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:00.525926] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C298-0-0] [lt=22] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326260017888003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326260017888003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:00.527680] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=1858, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:00.527801] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=118] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9112) [2025-01-08 00:51:00.527824] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=228, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:00.527854] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=13, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:00.528499] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=131] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8595, offset=2438537}, end_cursor:ObLogCursor{file_id=1, log_id=8596, offset=2438842}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89473c050, len:2679, log_data_len:305, seq:8595, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:00.530137] INFO [LIB] stat (utility.h:1210) [58356][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BEF-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:00.530279] INFO [LIB] stat (utility.h:1210) [58356][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BEF-0-0] [lt=141] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4597, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:00.530296] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58356][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BEF-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=110, avg_log_submit_to_flush_time=2256, avg_log_submit_to_slide_time=4486) [2025-01-08 00:51:00.531309] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9112, hit_cnt=14, read_cnt=14, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:00.532575] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=105] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326259713122004, v:0}, new_max_version={val:1736326259713122004, v:0}, last_min_version={val:1736326259509891004, v:0}, last_max_version={val:1736326259509891004, v:0}, total_time_us=10250, delta_us=819450) [2025-01-08 00:51:00.532667] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=88] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326259713122004, v:0}, min_version={val:1736326259713122004, v:0}, max_version={val:1736326259713122004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:00.534798] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E45-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:00.534851] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E45-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326260433701000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326260433701000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:00.542172] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=107] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.544254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.546386] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8596, offset=2438842}, end_cursor:ObLogCursor{file_id=1, log_id=8597, offset=2439139}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894740050, len:2374, log_data_len:297, seq:8596, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:00.551678] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E45-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:00.551741] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E45-0-0] [lt=42] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326260433701000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326260433701000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:00.552149] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=36] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:00.554477] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58374][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3889-0-0] [lt=61] init point thread id with(&point=0x5572d50e2ac0, idx_=1030, point=[thread id=58374, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58374) [2025-01-08 00:51:00.556786] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.561846] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1736326260561843, bytes: 0, bw: 0.000000 MB/s, add_ts: 1010329, add_bytes: 0 [2025-01-08 00:51:00.561871] INFO pn_print_diag_info (group.c:775) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-01-08 00:51:00.566666] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17793, miss_cnt:0, cache_read_size:2176627, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:00.572582] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8597, offset=2439139}, end_cursor:ObLogCursor{file_id=1, log_id=8598, offset=2439436}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894744050, len:2077, log_data_len:297, seq:8597, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:00.573980] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8598, offset=2439436}, end_cursor:ObLogCursor{file_id=1, log_id=8599, offset=2439733}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894748050, len:1780, log_data_len:297, seq:8598, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:00.577957] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=35] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:00.589298] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4104, avg_group_log_size=373) [2025-01-08 00:51:00.589395] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=89] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4104) [2025-01-08 00:51:00.589417] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:00.593560] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:00.593910] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD2-0-0] [lt=30] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:00.593964] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD2-0-0] [lt=44] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=56) [2025-01-08 00:51:00.595016] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=45] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:00.600842] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1609BCBD-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326260599789000, start_id=1736326260599789000, end_id=1736326260599789001) [2025-01-08 00:51:00.601545] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=142] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:51:00.607972] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=110] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:00.619659] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:00.622284] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:00.622966] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:00.622998] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:00.623017] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:00.623032] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:00.626787] INFO [LIB] stat (utility.h:1210) [58371][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9080-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=26, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:00.626812] INFO [LIB] stat (utility.h:1210) [58371][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9080-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5825, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:00.626823] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58371][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9080-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=127, avg_log_submit_to_flush_time=4155, avg_log_submit_to_slide_time=5698) [2025-01-08 00:51:00.627452] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8437-0-0] [lt=52] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:00.628335] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:00.628373] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7186, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:00.656965] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.661899] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:00.663978] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58371][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3789-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=2017, avg_log_submit_to_slide_time=20621) [2025-01-08 00:51:00.664136] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=198] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:00.686677] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5196636257, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:00.686721] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:00.686735] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a02b0) [2025-01-08 00:51:00.686745] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5196636257, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:00.703047] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EDA-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:00.725298] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3301, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:00.725341] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=407, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:00.727526] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-01-08 00:51:00.733284] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085149-0-0] [lt=48] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:00.736047] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EDB-0-0] [lt=54] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:00.750956] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EDC-0-0] [lt=49] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:00.751708] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EE-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:00.751752] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EE-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326260653066000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326260653066000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:00.757095] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=96][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.762147] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EDD-0-0] [lt=42] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:00.763517] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=79] failed to get result(ret=-4008) [2025-01-08 00:51:00.763633] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=37] check new jobs(ret=0, tenant_id=1, is_oracle_tenant=false, job_infos=[{tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326178662155, start_date_:1736323478339648, last_date_:1736326178341674, this_date_:0, next_date_:1736327078339648, end_date_:64060560000000000, total_:8578451, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736408978339542, last_date_:0, this_date_:0, next_date_:1736408978339542, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:51:00.763788] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=151] check all tenants(ret=0) [2025-01-08 00:51:00.768522] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=38] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:00.769419] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=31] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:00.775718] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EDF-0-0] [lt=11] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:00.782964] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=84] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:00.788415] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EE0-0-0] [lt=39] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:00.795484] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D020-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:00.798162] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EE1-0-0] [lt=88] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:00.808460] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=90] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:00.811229] INFO [LIB] stat (utility.h:1210) [58371][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44C5-0-0] [lt=29] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:00.814393] INFO [LIB] stat (utility.h:1210) [58371][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC843B-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=20, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:00.814431] INFO [LIB] stat (utility.h:1210) [58371][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC843B-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=37556, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:00.820087] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:00.823568] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=23] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:00.825158] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=34] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:00.829710] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58379][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:00.829813] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29] worker thread created(tenant_->id()=1002, group_id_=1, token=5) [2025-01-08 00:51:00.830030] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=33] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:00.854531] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EE2-0-0] [lt=95] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:00.856444] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EE3-0-0] [lt=36] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:00.857312] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.859014] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=38] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1003}, {id:1}, {id:1002}]) [2025-01-08 00:51:00.864732] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC378B-0-0] [lt=84] init point thread id with(&point=0x5572d50e2c00, idx_=1035, point=[thread id=58379, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58379) [2025-01-08 00:51:00.864896] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC378B-0-0] [lt=38] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:00.864954] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC378B-0-0] [lt=32] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:51:00.893672] INFO pn_print_diag_info (group.c:755) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO client:0x2ad88e1fca58_"192.168.159.138:2882"_"192.168.159.138:2882"_146_1736322505625436_1, write_queue=0/0, write=69414/71810919, read=69414/15894134, doing=0, done=69415, write_time=90897954, read_time=104342, process_time=110413 [2025-01-08 00:51:00.893704] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=118] PNIO [ratelimit] time: 1736326260893699, bytes: 869161172, bw: 0.268670 MB/s, add_ts: 1007719, add_bytes: 283895 [2025-01-08 00:51:00.893757] INFO pn_print_diag_info (group.c:755) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=78] PNIO client:0x2ad88e162048_"192.168.159.137:2882"_"192.168.159.137:2882"_130_1736322504706680_1, write_queue=0/0, write=16387/16556058, read=16386/4022791, doing=0, done=16387, write_time=17873417, read_time=1633, process_time=2977 [2025-01-08 00:51:00.893778] INFO pn_print_diag_info (group.c:755) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO client:0x2ad88e086a98_"192.168.159.139:2882"_"192.168.159.139:2882"_124_1736322504683494_1, write_queue=0/0, write=69309/60820198, read=69309/16696477, doing=0, done=69310, write_time=87169913, read_time=231767, process_time=238006 [2025-01-08 00:51:00.893792] INFO pn_print_diag_info (group.c:768) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO server:0x2ad88e1fc048_"192.168.159.139:45074"_144_1736322505188012, write_queue=0/0, write=102161/53927647, read=102161/108089264, doing=0, done=102161, write_time=123131243, read_time=2710198, process_time=2718026 [2025-01-08 00:51:00.893800] INFO pn_print_diag_info (group.c:755) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=87] PNIO client:0x2ad88e1fe048_"192.168.159.139:2882"_"192.168.159.139:2882"_148_1736322505625884_1, write_queue=0/0, write=69184/70158910, read=69184/15314693, doing=0, done=69184, write_time=89084988, read_time=14065, process_time=19947 [2025-01-08 00:51:00.893810] INFO pn_print_diag_info (group.c:768) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO server:0x2ad88e1fa048_"192.168.159.138:42010"_140_1736322504947415, write_queue=0/0, write=105234/52657874, read=105234/81068318, doing=0, done=105234, write_time=123701906, read_time=1099452, process_time=1107761 [2025-01-08 00:51:00.893825] INFO pn_print_diag_info (group.c:755) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO client:0x2ad88e162a98_"192.168.159.138:2882"_"192.168.159.138:2882"_131_1736322504706708_1, write_queue=0/0, write=69296/60282765, read=69296/15443191, doing=0, done=69297, write_time=85299504, read_time=48415, process_time=55534 [2025-01-08 00:51:00.893840] INFO pn_print_diag_info (group.c:755) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO client:0x2ad88e004808_"192.168.159.137:2882"_"192.168.159.137:2882"_122_1736322504678620_1, write_queue=0/0, write=16630/15472571, read=16630/4111697, doing=0, done=16632, write_time=18741694, read_time=2063, process_time=3152 [2025-01-08 00:51:00.893859] INFO pn_print_diag_info (group.c:768) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO server:0x2ad88e1faa58_"192.168.159.139:45058"_142_1736322505176841, write_queue=0/0, write=102197/47486232, read=102197/77709611, doing=0, done=102197, write_time=128744412, read_time=908664, process_time=916973 [2025-01-08 00:51:00.893874] INFO pn_print_diag_info (group.c:768) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO server:0x2ad88e1a8a58_"192.168.159.138:41998"_138_1736322504924846, write_queue=0/0, write=105032/52237396, read=105032/81253674, doing=0, done=105032, write_time=129787900, read_time=743375, process_time=751939 [2025-01-08 00:51:00.893888] INFO pn_print_diag_info (group.c:768) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO server:0x2ad88e005258_"192.168.159.137:41266"_123_1736322504679850, write_queue=0/0, write=16630/4111697, read=16630/15472571, doing=0, done=16630, write_time=17905816, read_time=3345, process_time=4441 [2025-01-08 00:51:00.893899] INFO pn_print_diag_info (group.c:775) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO client_send:155113/145915336, client_queue_time=171904838, cnt=3, server_send:223859/103835325, server_queue_time=150780188, cnt=3 [2025-01-08 00:51:00.893828] INFO pn_print_diag_info (group.c:768) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO server:0x2ad88e1a8048_"192.168.159.137:41278"_133_1736322504708541, write_queue=0/0, write=16386/4022791, read=16387/16556058, doing=0, done=16387, write_time=17642836, read_time=3708, process_time=4954 [2025-01-08 00:51:00.893977] INFO pn_print_diag_info (group.c:775) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=148] PNIO client_send:155112/149187853, client_queue_time=192650571, cnt=3, server_send:223782/110608534, server_queue_time=143507442, cnt=3 [2025-01-08 00:51:00.896729] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=75] ====== check clog disk timer task ====== [2025-01-08 00:51:00.896789] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=50] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:00.897550] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.899376] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.899531] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.899587] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:00.909604] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=39] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18655744}, readable_log_tail:{lsn:18655744}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48449024, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:00.909736] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=130] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:00.909750] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2154292}, readable_log_tail:{lsn:2154292}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64950476, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:00.909863] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=111] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:00.909877] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] get_replay_process success(id={id:1002}, submitted_log_size=2187206, unsubmitted_log_size=2187206, replayed_log_size=2187206, unreplayed_log_size=0) [2025-01-08 00:51:00.909887] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] get_replay_process success(id={id:1003}, submitted_log_size=2176871, unsubmitted_log_size=2176871, replayed_log_size=2176871, unreplayed_log_size=0) [2025-01-08 00:51:00.909958] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:00.914104] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58379][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6D8-0-0] [lt=66] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:00.916953] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326260615562000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=301374, in_cluster_service=true, cluster_version={val:1736326260615562000, v:0}, min_cluster_version={val:1736326260615562000, v:0}, max_cluster_version={val:1736326260615562000, v:0}, get_cluster_version_err=0, cluster_version_delta=301374, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36354, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=515413923, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:00.928836] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=9273, avg_group_log_size=662) [2025-01-08 00:51:00.928903] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=60] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9274) [2025-01-08 00:51:00.929335] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=28] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5197076320, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:51:00.930882] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=57] ====== check clog disk timer task ====== [2025-01-08 00:51:00.930918] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=29] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:00.932318] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=42] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5197076320, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:51:00.934833] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:00.935860] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:00.954788] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=917] no need to update min/max version(new_version={val:1736326260559622000, v:0}, min_version={val:1736326260559622000, v:0}, max_version={val:1736326260559622000, v:0}) [2025-01-08 00:51:00.957396] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=91][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:00.964875] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC378D-0-0] [lt=165] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:00.964913] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC378D-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=32250, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:00.964924] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC378D-0-0] [lt=8][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17885, log_task={header:{begin_lsn:{lsn:2176871}, end_lsn:{lsn:2176993}, log_id:17885, min_scn:{val:1736326260714990000, v:0}, max_scn:{val:1736326260714990000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2176749}, prev_proposal_id:1, committed_end_lsn:{lsn:2176871}, data_checksum:940009244, accum_checksum:1698274773, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326260864793, freeze_ts:1736326260864793, submit_ts:1736326260864931, flushed_ts:1736326260869034, gen_to_freeze cost time:0, gen_to_submit cost time:138, submit_to_flush cost time:4103}, fs_cb_begin_ts=1736326260964859, log_life_time=100066) [2025-01-08 00:51:00.977684] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=30] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:00.983936] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:00.998612] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C8C-0-0] [lt=64][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:01.001377] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2512, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:01.001428] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=307, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:01.001468] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=36, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:01.002985] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C8C-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:01.003471] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B12-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:01.003506] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B12-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5459, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:01.003519] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B12-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=160, avg_log_submit_to_flush_time=2859, avg_log_submit_to_slide_time=5298) [2025-01-08 00:51:01.003573] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=202] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3983, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:01.009526] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=82] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:01.010740] INFO [SERVER] refresh_avail_memory (ob_table_load_assigned_memory_manager.cpp:117) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] ObTableLoadAssignedMemoryManager::refresh_avail_memory(MTL_ID()=1002, avail_memory=161061273, chunk_count_=0, avail_sort_memory_=161061273, avail_memory_=161061273) [2025-01-08 00:51:01.012140] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58381][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:01.012174] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44C9-0-0] [lt=35] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:01.012617] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread created(tenant_->id()=1002, group_id_=1, token=6) [2025-01-08 00:51:01.013689] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:01.015167] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6DA-0-0] [lt=106] init point thread id with(&point=0x5572d50e2c80, idx_=1037, point=[thread id=58381, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58381) [2025-01-08 00:51:01.020433] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4569) [2025-01-08 00:51:01.022039] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:01.023843] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:01.026912] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=56] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:01.028322] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:01.028390] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:01.033020] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D33-0-0] [lt=36] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:01.037675] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9273) [2025-01-08 00:51:01.057539] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=100] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:01.057582] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.077175] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB74-0-0] [lt=24][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193765991007, cache_obj->added_lc()=false, cache_obj->get_object_id()=13580, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:01.078410] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=73] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:01.093756] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=96] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:01.101775] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:51:01.103757] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:01.103810] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44955, [delay/us]:prepare:8, schedule:642, submit:508, rt:1225, total:2385) [2025-01-08 00:51:01.103880] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=62] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:01.104026] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=93] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44948, [delay/us]: prepare:8, schedule:642, submit:508, device:1225, total:2385) [2025-01-08 00:51:01.104044] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:01.104098] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4915, iops:15, bw:73551, [delay/us]:prepare:7, schedule:145, submit:462, rt:1455, total:2070) [2025-01-08 00:51:01.104153] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=50] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:01.104255] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4915, avg_iops:15, avg_bw:73543, [delay/us]: prepare:7, schedule:145, submit:462, device:1455, total:2070) [2025-01-08 00:51:01.104270] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:01.104295] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16342, [delay/us]:prepare:6, schedule:162, submit:1, rt:1331, total:1500) [2025-01-08 00:51:01.104317] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4306, iops:39, bw:167514, [delay/us]:prepare:19, schedule:190, submit:501, rt:1528, total:2239) [2025-01-08 00:51:01.104327] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:01.104359] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16342, [delay/us]: prepare:6, schedule:162, submit:1, device:1331, total:1500) [2025-01-08 00:51:01.104374] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4306, avg_iops:39, avg_bw:167513, [delay/us]: prepare:19, schedule:190, submit:501, device:1528, total:2239) [2025-01-08 00:51:01.104385] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:01.104403] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4085, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:9438, total:9452) [2025-01-08 00:51:01.104427] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:01.104453] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:9438, total:9452) [2025-01-08 00:51:01.104472] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:01.104487] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:01.104495] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:01.104503] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:01.104511] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=279kB/s, net_out=227kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:01.112115] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1764, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:01.112151] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=266, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:01.112286] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=76, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:01.117656] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B4BBA92-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1736326261117236000, start_id=1736326261117236000, end_id=1736326261117236001) [2025-01-08 00:51:01.128948] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:01.157625] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.162083] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:01.163870] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC389-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:01.165869] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2181333, this={dio_aligned_buf:{buf_write_offset:2383, buf_padding_size:1713, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:23858, truncate_used_ts:5430}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2179072, buf_len=122, total_write_size=12050, total_write_size_after_dio=417792, ob_pwrite_used_ts=352963, count=99, each_pwrite_cost=3565) [2025-01-08 00:51:01.165921] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1210) [2025-01-08 00:51:01.168129] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=324, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:01.168155] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=7, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:01.188521] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=143] ====== check clog disk timer task ====== [2025-01-08 00:51:01.188635] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=72] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:01.199733] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=115, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:01.210023] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:01.212123] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=98] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=534769, clean_num=31457) [2025-01-08 00:51:01.214647] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17885, miss_cnt:0, cache_read_size:2187694, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:01.218059] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61] no need to update min/max version(new_version={val:1736326260816335000, v:0}, min_version={val:1736326260816335000, v:0}, max_version={val:1736326260816335000, v:0}) [2025-01-08 00:51:01.222328] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=109] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:01.224501] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1765, avg_group_log_size=160) [2025-01-08 00:51:01.224541] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1765) [2025-01-08 00:51:01.225466] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=98] gc stale ls task succ [2025-01-08 00:51:01.226496] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:01.229333] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:01.231810] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:01.231839] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:01.236366] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=102, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:01.236888] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D16EBBD9E-0-0] [lt=22] get number(ret=0, service_type_=0, range=1, base_id=1736326261235251000, start_id=1736326261235251000, end_id=1736326261235251001) [2025-01-08 00:51:01.239516] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C2BB5EB-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1736326261239224000, start_id=1736326261239224000, end_id=1736326261239224001) [2025-01-08 00:51:01.242613] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:01.242980] WDIAG read_multi_row (ob_table_access_helper.h:238) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:01.243809] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=25] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:01.245305] INFO delete_row (ob_table_access_helper.h:339) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326261243002) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326261243002) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:01.245400] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=41] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:01.245459] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:51:01.245510] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:51:01.248448] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8704C-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:01.248616] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:01.248632] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:01.255255] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=44] no need to update min/max version(new_version={val:1736326260766551000, v:0}, min_version={val:1736326260766551000, v:0}, max_version={val:1736326260766551000, v:0}) [2025-01-08 00:51:01.257796] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.263280] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=49] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:01.263307] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:01.263392] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E5-0-0] [lt=15] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:01.263415] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E5-0-0] [lt=18] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:01.264536] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3793-0-0] [lt=89] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1210) [2025-01-08 00:51:01.266286] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1706, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:01.266390] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=94] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=524, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:01.266458] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=60, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:01.282310] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326257142695, last_first_register_time_us=1736324454612308, register_time_us=1736326257142695, curr_time_us=1736326261282306) [2025-01-08 00:51:01.282405] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=58] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326261282370, last_submit_register_req_time_us=1736326261282370, register_parent_reason=1) [2025-01-08 00:51:01.282424] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:01.283692] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=84] gc stale ls task succ [2025-01-08 00:51:01.285406] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF45-0-0] [lt=25] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326261386780, keepalive_ts:-1}) [2025-01-08 00:51:01.285444] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF45-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326261386780, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:01.294890] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:01.304562] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA3-0-0] [lt=17] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:01.304604] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA3-0-0] [lt=39] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:01.312666] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=23] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:01.313379] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:300K/70:174702 disk:300K/70:174702 t1m2:48K/11:225154 t1m2g0:48K/11:225154 t1002m2:180K/43:182816 t1002m2g0:180K/43:182816 t1001m2:72K/16:118212 t1001m2g0:72K/16:118212 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:01.320420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.320459] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.323592] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185418-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:01.328658] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] get number(ret=0, service_type_=0, range=1, base_id=1736326261328063000, start_id=1736326261328063000, end_id=1736326261328063001) [2025-01-08 00:51:01.333922] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58299][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBF-0-0] [lt=34] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326261435468, keepalive_ts:-1}) [2025-01-08 00:51:01.333969] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58299][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBF-0-0] [lt=46] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326261435468, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:01.340106] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:01.342180] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58356][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:01.348421] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=87] current monitor number(seq_=-1) [2025-01-08 00:51:01.357898] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.408786] INFO [STORAGE.COMPACTION] schedule_all_tablets_medium (ob_tenant_tablet_scheduler.cpp:1597) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58542B-0-0] [lt=43] start schedule all tablet merge(merge_version=1) [2025-01-08 00:51:01.409065] INFO [STORAGE] get_adaptive_merge_reason (ob_partition_merge_policy.cpp:1719) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58542B-0-0] [lt=31] Check tablet adaptive merge reason(ret=0, ls_id={id:1}, tablet_id={id:60379}, reason=0, tablet_analyzer={tablet_stat:{ls_id:1, tablet_id:60379, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, total_tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_small_tenant:false, boost_factor:"1.000000000000000000e+00"}, crazy_medium_flag=false) [2025-01-08 00:51:01.412621] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=91] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:01.415601] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:51:01.416279] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2187872}, scn={val:1736326261012826003, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:01.416319] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:33, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326261412908, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2187816}, next_to_submit_scn_:{val:1736326261012826003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:56, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2187816}, end_lsn:{lsn:2187928}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2188050}, readable_log_tail:{lsn:2188050}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64916718, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326261012826003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326261012826003, v:0}, data_checksum:3634441214, flag:62842}}, init_mode_version:0, accumulate_checksum:1346838580, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326261012826003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17887, miss_cnt:0, cache_read_size:2187928, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2187872}, scn_:{val:1736326261012826003, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326261426951, first_handle_ts_:1736326261416317, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad968204150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:51:01.416571] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:01.416594] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70075580, tenant_id=1001) [2025-01-08 00:51:01.416615] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:01.417132] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=99] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324455577817000) [2025-01-08 00:51:01.417153] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=18] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2187872}, scn_:{val:1736326261012826003, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326261426951, first_handle_ts_:1736326261417118, replay_cost_:32, retry_cost_:-1, read_log_buf_:0x2ad968204160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326261412908, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2187928}, next_to_submit_scn_:{val:1736326261012826004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2187816}, end_lsn:{lsn:2187928}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2188050}, readable_log_tail:{lsn:2188050}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64916718, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326261012826003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326261012826003, v:0}, data_checksum:3634441214, flag:62842}}, init_mode_version:0, accumulate_checksum:1346838580, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326261012826003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17887, miss_cnt:0, cache_read_size:2187928, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=7, ret=0) [2025-01-08 00:51:01.422575] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:01.424464] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=90, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:01.426063] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2976, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:01.426092] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1732, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:01.426125] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:01.429371] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=36] ====== tenant freeze timer task ====== [2025-01-08 00:51:01.435868] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=27] ts response task statistics(alloc_count=59670, free_count=29834) [2025-01-08 00:51:01.436628] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=85] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:01.440754] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58383][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:01.440841] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58383][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2BFE-0-0] [lt=54] init point thread id with(&point=0x5572d50e2d00, idx_=1039, point=[thread id=58383, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58383) [2025-01-08 00:51:01.441028] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:51:01.458131] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=103][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.458615] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.459261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.465066] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=61] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=658198528, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=415543296, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:01.465150] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=46] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:01.465170] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2810, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:01.465291] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1112293376, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2108932096, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:01.465350] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:01.465365] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:01.484352] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=76] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:01.484417] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:01.497309] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46503][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB76-0-0] [lt=28][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193766411142, cache_obj->added_lc()=false, cache_obj->get_object_id()=13581, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:01.501025] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=5159) [2025-01-08 00:51:01.516252] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6E5-0-0] [lt=16] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=22, avg_log_submit_to_flush_time=1957, avg_log_submit_to_slide_time=19313) [2025-01-08 00:51:01.521472] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.531961] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=2259, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:01.531998] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=35] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9596) [2025-01-08 00:51:01.532019] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=164, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:01.532164] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=40, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:01.532811] INFO [LIB] stat (utility.h:1210) [58299][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3898-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=12, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:01.532846] INFO [LIB] stat (utility.h:1210) [58299][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3898-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=4425, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:01.532860] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58299][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3898-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=189, avg_log_gen_to_submit_time=296, avg_log_submit_to_flush_time=2464, avg_log_submit_to_slide_time=4128) [2025-01-08 00:51:01.532926] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9596, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:01.533402] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326261012826000, v:0}, new_max_version={val:1736326261012826000, v:0}, last_min_version={val:1736326260628722002, v:0}, last_max_version={val:1736326260628722002, v:0}, total_time_us=6158, delta_us=520574) [2025-01-08 00:51:01.533435] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326261012826000, v:0}, min_version={val:1736326261012826000, v:0}, max_version={val:1736326261012826000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:01.542384] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=79] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.543306] INFO pn_ratelimit (group.c:653) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=12] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:01.544271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.558281] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.566734] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1736326261566730, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004887, add_bytes: 0 [2025-01-08 00:51:01.567692] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=136] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17803, miss_cnt:0, cache_read_size:2177847, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:01.570121] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C299-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:01.570245] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C299-0-0] [lt=107] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326261012826005, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326261012826005, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:01.577609] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1A79C299-0-0] [lt=97] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.671243333816528320e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=383, Throttle Trigger(MB)=230, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=8.103636458226476407e+02) [2025-01-08 00:51:01.579527] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8599, offset=2439733}, end_cursor:ObLogCursor{file_id=1, log_id=8600, offset=2440038}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89474c050, len:1483, log_data_len:305, seq:8599, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:01.580049] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E46-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:01.580111] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E46-0-0] [lt=47] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326261435556000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326261435556000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:01.585868] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E46-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1736326261585261000, start_id=1736326261585261000, end_id=1736326261585261001) [2025-01-08 00:51:01.588093] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=96] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:01.595681] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:01.595932] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=46] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8600, offset=2440038}, end_cursor:ObLogCursor{file_id=1, log_id=8601, offset=2440335}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894750050, len:1178, log_data_len:297, seq:8600, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:01.597161] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E46-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:01.597255] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E46-0-0] [lt=72] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326261435556000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326261435556000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:01.601919] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=6] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:51:01.610348] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=79] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8601, offset=2440335}, end_cursor:ObLogCursor{file_id=1, log_id=8602, offset=2440632}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894754050, len:881, log_data_len:297, seq:8601, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:01.612265] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=102] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8602, offset=2440632}, end_cursor:ObLogCursor{file_id=1, log_id=8603, offset=2440929}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894758050, len:584, log_data_len:297, seq:8602, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:01.613019] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=107] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:01.614934] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1454, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:01.622872] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:01.624755] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=39, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:01.624789] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:01.624808] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:01.624823] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:01.627546] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get_min_unreplayed_log_info(lsn={lsn:2188294}, scn={val:1736326261513350001, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326261614729, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2188294}, next_to_submit_scn_:{val:1736326261513350001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2188172}, end_lsn:{lsn:2188294}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2188294}, readable_log_tail:{lsn:2188294}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64916474, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326261513350000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326261513350000, v:0}, data_checksum:3348837161, flag:28147}}, init_mode_version:0, accumulate_checksum:2896429349, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326261513350000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17890, miss_cnt:0, cache_read_size:2188294, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:01.628549] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A909D-0-0] [lt=46] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=46, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:01.628574] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A909D-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6596, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:01.628584] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A909D-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=118, avg_log_submit_to_flush_time=4698, avg_log_submit_to_slide_time=6478) [2025-01-08 00:51:01.629361] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A909E-0-0] [lt=96] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:01.629498] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A909E-0-0] [lt=91] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:01.629512] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A909E-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5713, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:01.629904] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58384][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:01.630035] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:01.658579] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.662253] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=34] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:01.662724] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:51:01.662517] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58385][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:01.669600] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6E8-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=29, avg_log_submit_to_flush_time=1733, avg_log_submit_to_slide_time=12680) [2025-01-08 00:51:01.669710] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=97] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:01.689393] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5199157327, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:01.689458] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:01.689475] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a1eb0) [2025-01-08 00:51:01.689485] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5199157327, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:01.698440] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A8A-0-0] [lt=51] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:01.700093] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=5159, avg_group_log_size=429) [2025-01-08 00:51:01.700123] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5159) [2025-01-08 00:51:01.700140] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:01.705981] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [50457][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=77] XA scheduler heartbeat task statistics(avg_time=1) [2025-01-08 00:51:01.718942] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=100] no need to update min/max version(new_version={val:1736326261319676000, v:0}, min_version={val:1736326261319676000, v:0}, max_version={val:1736326261319676000, v:0}) [2025-01-08 00:51:01.719044] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=95] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:01.730366] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3280, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:01.730457] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=86] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=168, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:01.734357] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=242] get_max_applied_scn(scn={val:1736326261435556001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18657265}, palf_committed_end_scn_:{val:1736326261435556002, v:0}, last_check_scn_:{val:1736326261722670000, v:0}, max_applied_cb_scn_:{val:1736326261435556001, v:0}}) [2025-01-08 00:51:01.734455] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=91] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18657810}, readable_log_tail:{lsn:18657810}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48446958, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:01.735315] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=166] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:01.747754] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=55] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:51:01.751398] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A8B-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:01.758964] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=124][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.768976] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:01.775478] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A8C-0-0] [lt=28] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:01.778519] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=34] failed to get result(ret=-4008) [2025-01-08 00:51:01.778681] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=45] check new jobs(ret=0, tenant_id=1002, is_oracle_tenant=false, job_infos=[{tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326232555425, start_date_:1736325329924481, last_date_:1736326229926861, this_date_:0, next_date_:1736327129924481, end_date_:64060560000000000, total_:6030537, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736410829924386, last_date_:0, this_date_:0, next_date_:1736410829924386, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:51:01.778834] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=150] check all tenants(ret=0) [2025-01-08 00:51:01.783316] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=77] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:01.798359] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D023-0-0] [lt=113][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:01.799522] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EF-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:01.799569] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5EF-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326261717746000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326261717746000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:01.805146] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A8F-0-0] [lt=12] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:01.812475] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44D9-0-0] [lt=81] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:01.813731] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:01.817525] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6ED-0-0] [lt=60] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=35, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:01.817622] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6ED-0-0] [lt=95] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=10847, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:01.818784] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A90-0-0] [lt=27] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:01.820773] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA389E-0-0] [lt=270] init point thread id with(&point=0x5572d50e2d80, idx_=1041, point=[thread id=58385, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58385) [2025-01-08 00:51:01.823508] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:01.824098] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=22] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:01.825711] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=162] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:01.828449] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:801) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D1718511A-0-0] [lt=25] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, location:{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, location:{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}]) [2025-01-08 00:51:01.828686] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=49] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:01.828713] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=19] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=206) [2025-01-08 00:51:01.828725] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:01.828753] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=20] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:01.828784] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=29] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=185) [2025-01-08 00:51:01.830572] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A91-0-0] [lt=22] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:01.832983] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A92-0-0] [lt=21] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:01.859149] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.859514] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=37] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}, {id:1003}]) [2025-01-08 00:51:01.864428] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A93-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:01.868585] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC379F-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:01.868637] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC379F-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:01.869463] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] [COMPACTION DAG STATUS] (start_time=1736326251859905, end_time=1736326261869461, dag_status={}) [2025-01-08 00:51:01.869505] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=39] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:01.869520] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:01.869528] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:01.869535] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:01.869604] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=68] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:01.869613] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:01.869620] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:01.869626] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:01.869632] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:01.869639] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869649] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869657] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869664] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869677] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869685] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869692] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869699] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869706] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869713] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869727] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869734] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869742] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869749] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869756] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869763] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869777] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869784] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869791] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869799] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869806] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869867] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869876] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869883] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869890] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869897] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869904] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869911] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869918] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869925] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869932] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869939] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869946] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869953] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869961] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869968] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869975] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.869982] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870012] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=29] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870024] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870032] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870039] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870048] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870056] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870153] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=97] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870165] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870223] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870232] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870240] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870248] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870256] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870264] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870272] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870280] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870287] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870295] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870303] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870311] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870319] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870327] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870335] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870343] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870351] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870359] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870366] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870375] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870382] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870390] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870396] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870404] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870411] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:01.870420] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:01.870426] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:01.870433] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:01.870438] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:01.870442] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:01.870447] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:01.870451] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:01.870455] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:01.870460] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:01.870469] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:01.870474] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:01.879465] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=35] LogLoopThread round_cost_time(us)(round_cost_time=19) [2025-01-08 00:51:01.896259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=183] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.897424] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.897955] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.898001] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:01.899638] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1736326261899635, bytes: 869383924, bw: 0.211179 MB/s, add_ts: 1005936, add_bytes: 222752 [2025-01-08 00:51:01.902766] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A94-0-0] [lt=26] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:01.908312] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8B-00062B2D194BB26A-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1736326261905242000, start_id=1736326261905242000, end_id=1736326261905242001) [2025-01-08 00:51:01.914379] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D197BB4CE-0-0] [lt=27] handle gts request statistics(total_rt=1898, total_cnt=288, avg_rt=6.567474048442906387e+00, avg_cnt=2.880000000000000071e+01) [2025-01-08 00:51:01.916260] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44DB-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1210) [2025-01-08 00:51:01.930023] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=9594, avg_group_log_size=564) [2025-01-08 00:51:01.930143] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=113] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9593) [2025-01-08 00:51:01.936740] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:01.937317] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2066, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:01.937855] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=47] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:01.945410] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C29C26E-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1736326261942997000, start_id=1736326261942997000, end_id=1736326261942997001) [2025-01-08 00:51:01.959326] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=116][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:01.967117] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A1-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:01.967164] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A1-0-0] [lt=46] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12668, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:01.984672] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:01.990263] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:01.992355] INFO [STORAGE.COMPACTION] add_event_and_diagnose (ob_medium_loop.cpp:203) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58542B-0-0] [lt=41] finish schedule all tablet merge(merge_version_=1, schedule_stats_={all_ls_weak_read_ts_ready:true, start_timestamp:0}, loop_cnt=0, tenant_merge_finish=true, is_scan_all_tablet_finish=true, schedule_tablet_cnt={schedule_dag_cnt:0, submit_clog_cnt:0, finish_cnt:710, loop_tablet_cnt:710, force_freeze_cnt:0, wait_rs_validate_cnt:0}, time_guard=GET_TABLET=12.72ms|INIT_TABLET_STATUS=59.42ms|SCHEDULE_NEXT_MEDIUM=4.45ms|SCHEDULE_TABLET_EXECUTE=506.70ms|) [2025-01-08 00:51:01.992459] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:95) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58542B-0-0] [lt=97] MediumLoopTask(cost_ts=583745) [2025-01-08 00:51:01.992572] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D18539A-0-0] [lt=37] MediumCheckTask(cost_ts=8) [2025-01-08 00:51:02.008692] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1738, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:02.008950] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=224] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=303, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:02.010422] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=25, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:02.012514] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58386][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:02.012566] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:51:02.012694] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58386][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67B5-0-0] [lt=143] init point thread id with(&point=0x5572d50e2dc0, idx_=1042, point=[thread id=58386, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58386) [2025-01-08 00:51:02.012913] INFO [LIB] stat (utility.h:1210) [58386][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67B5-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:02.012928] INFO [LIB] stat (utility.h:1210) [58386][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67B5-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3898, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:02.013305] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C8D-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:02.014727] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C8D-0-0] [lt=37][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:02.015314] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:02.022013] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:51:02.026295] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=111] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1148144, replace_num=15728) [2025-01-08 00:51:02.026335] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:02.027043] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:02.027172] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58387][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:02.029622] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2066) [2025-01-08 00:51:02.029821] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:51:02.034409] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90A9-0-0] [lt=170] init point thread id with(&point=0x5572d50e2d40, idx_=1040, point=[thread id=58384, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58384) [2025-01-08 00:51:02.034764] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84226175}], scn_array=[cnt:1, {val:1736326262018982001, v:0}], curr_size=586, accum_size=4996, time_cost=12640) [2025-01-08 00:51:02.034807] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4996) [2025-01-08 00:51:02.035439] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67B6-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=3160, avg_log_submit_to_slide_time=4603) [2025-01-08 00:51:02.035806] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5582, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:02.036068] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] worker thread created(tenant_->id()=1001, group_id_=1, token=8) [2025-01-08 00:51:02.059403] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.063345] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=77] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10140) [2025-01-08 00:51:02.068459] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B92-0-0] [lt=86][errcode=-4006] not init(ret=-4006) [2025-01-08 00:51:02.068544] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:480) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B92-0-0] [lt=31] finish scan map holding items(scan_cnt=0) [2025-01-08 00:51:02.068675] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B93-0-0] [lt=14] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:02.069103] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A3-0-0] [lt=65][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17896, log_task={header:{begin_lsn:{lsn:2178213}, end_lsn:{lsn:2178335}, log_id:17896, min_scn:{val:1736326261717746002, v:0}, max_scn:{val:1736326261717746002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2178091}, prev_proposal_id:1, committed_end_lsn:{lsn:2178213}, data_checksum:940009244, accum_checksum:2955228656, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326261966994, freeze_ts:1736326261966994, submit_ts:1736326261967007, flushed_ts:1736326261968389, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:1382}, fs_cb_begin_ts=1736326262069044, log_life_time=102050) [2025-01-08 00:51:02.071883] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=45] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:02.071922] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=86) [2025-01-08 00:51:02.088441] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:02.095998] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:02.100950] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [46506][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16D9BA1F-0-0] [lt=40] sql stat record not found(ret=-5138, key={sql_id:"E268830A30994BFD0285516859916A6", plan_hash:3290318591324336132, source_addr:"192.168.159.138:41998"}) [2025-01-08 00:51:02.103424] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:51:02.110630] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:02.110685] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:12, bw:56952, [delay/us]:prepare:12, schedule:231, submit:236, rt:1675, total:2156) [2025-01-08 00:51:02.110701] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:02.110757] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:12, avg_bw:56960, [delay/us]: prepare:12, schedule:231, submit:236, device:1675, total:2156) [2025-01-08 00:51:02.110771] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:02.110810] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:17, bw:77305, [delay/us]:prepare:7, schedule:606, submit:159, rt:1934, total:2707) [2025-01-08 00:51:02.110820] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:02.110855] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:77313, [delay/us]: prepare:7, schedule:606, submit:159, device:1934, total:2707) [2025-01-08 00:51:02.110866] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:02.110900] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16276, [delay/us]:prepare:3, schedule:99, submit:0, rt:2058, total:2162) [2025-01-08 00:51:02.110918] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:705, bw:2889114, [delay/us]:prepare:1, schedule:173, submit:0, rt:364, total:540) [2025-01-08 00:51:02.110942] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4195, iops:41, bw:170903, [delay/us]:prepare:5, schedule:328, submit:211, rt:1788, total:2333) [2025-01-08 00:51:02.110955] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=176, ops=10, iops=186, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:02.110997] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:706, avg_bw:2888982, [delay/us]: prepare:1, schedule:173, submit:0, device:364, total:540) [2025-01-08 00:51:02.111006] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16275, [delay/us]: prepare:3, schedule:99, submit:0, device:2058, total:2162) [2025-01-08 00:51:02.111021] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4196, avg_iops:41, avg_bw:170896, [delay/us]: prepare:5, schedule:328, submit:211, device:1788, total:2333) [2025-01-08 00:51:02.111031] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:02.111051] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4068, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:597, total:610) [2025-01-08 00:51:02.111075] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:02.111102] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4068, [delay/us]: prepare:0, schedule:0, submit:600000000, device:597, total:610) [2025-01-08 00:51:02.111120] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:02.111135] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:02.111216] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=79] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:02.111225] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:02.111233] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=302kB/s, net_out=231kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:02.113672] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44DF-0-0] [lt=50] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=12, avg_log_batch_cnt=0, total_group_log_size=1454, avg_group_log_size=121) [2025-01-08 00:51:02.115087] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2553, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:02.115141] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=329, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:02.115280] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=64, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:02.121373] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185790-0-0] [lt=115][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193767035204, cache_obj->added_lc()=false, cache_obj->get_object_id()=13583, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:02.122356] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185791-0-0] [lt=48][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:02.125018] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185793-0-0] [lt=36][errcode=-4201] hashtable not init [2025-01-08 00:51:02.131560] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:02.159452] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=87][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.162599] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=99] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:02.205103] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=72, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:02.208447] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15] ====== traversal_flush timer task ====== [2025-01-08 00:51:02.208585] INFO [MDS] flush (mds_table_mgr.cpp:195) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485369-0-0] [lt=45] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326261935253001, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326270087570000, latest_log_ts:{val:1736326253487654001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.208752] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485369-0-0] [lt=131] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326256925355001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.208897] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485369-0-0] [lt=96] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326256992205000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.208989] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485369-0-0] [lt=63] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326256709695004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.209164] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485369-0-0] [lt=168] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:02.215962] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:02.220485] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:02.224561] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:02.225757] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=80] gc stale ls task succ [2025-01-08 00:51:02.226043] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:02.226139] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326254278994000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:02.226755] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:02.227086] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2188904}], scn_array=[cnt:1, {val:1736326262002109001, v:0}], curr_size=122, accum_size=244, time_cost=10821) [2025-01-08 00:51:02.230105] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=67] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:02.230141] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] get_min_unreplayed_log_info(lsn={lsn:2178335}, scn={val:1736326261717746003, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326262069048, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2178335}, next_to_submit_scn_:{val:1736326261717746003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2178213}, end_lsn:{lsn:2178335}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2178457}, readable_log_tail:{lsn:2178457}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64926311, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326261717746002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326261717746002, v:0}, data_checksum:2124447694, flag:37465}}, init_mode_version:0, accumulate_checksum:2955228656, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326261717746002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17807, miss_cnt:0, cache_read_size:2178335, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:02.231324] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:02.231374] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:02.231616] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE810-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326262231221000, start_id=1736326262231221000, end_id=1736326262231221001) [2025-01-08 00:51:02.231937] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:51:02.231962] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:02.236300] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17896, miss_cnt:0, cache_read_size:2189026, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:02.236944] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=101] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=201, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:02.236969] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:02.238103] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:02.238155] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:02.240381] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=72, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:02.257176] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=17] ====== checkpoint timer task ====== [2025-01-08 00:51:02.258567] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=88] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:02.258686] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=117] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:02.258803] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=69] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326270087570000, latest_log_ts:{val:1736326253487654001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.258915] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=107] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:02.258945] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:02.258967] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] get rec log scn(service_type_=0, rec_log_ts={val:1736326253487654001, v:0}) [2025-01-08 00:51:02.258979] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:02.258998] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:02.259020] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:02.259034] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:02.259056] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:02.259076] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:02.259737] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87052-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:02.259797] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:02.259809] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:02.259967] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:02.260001] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=34] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:02.260084] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=17] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326256925355001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.260149] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=62] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:02.260161] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:02.260279] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=112] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:02.260303] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=20] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:02.260319] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2156000}, scn={val:1736326261935253001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:02.260442] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.262914] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=51] get number(ret=0, service_type_=0, range=1, base_id=1736326262262761000, start_id=1736326262262761000, end_id=1736326262262761001) [2025-01-08 00:51:02.263597] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=5931) [2025-01-08 00:51:02.266390] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=26] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8603, offset=2440929}, end_cursor:ObLogCursor{file_id=1, log_id=8604, offset=2441226}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89475c050, len:4383, log_data_len:297, seq:8603, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:02.266518] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326261935253001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:02.266616] INFO [SHARE] add_event (ob_event_history_table_operator.h:266) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=30] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1736326262266549), 'checkpoint', 'update_clog_checkpoint', 'tenant_id', 1002, 'ls_id', 1001, 'checkpoint_scn', '{val:1736326261935253001, v:0}', 'service_type', 'MAX_DECIDED_SCN', 'update_clog_checkpoint_times', 120, '', '192.168.159.137', 2882)) [2025-01-08 00:51:02.266711] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:02.266745] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=33] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:02.267617] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:02.267655] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=38] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:02.267746] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=64] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326256992205000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.267819] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=71] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:02.267835] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:02.267853] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:02.267868] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:02.267883] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2189026}, scn={val:1736326262002109001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:02.272633] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58387][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38A5-0-0] [lt=163] init point thread id with(&point=0x5572d50e2e00, idx_=1043, point=[thread id=58387, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58387) [2025-01-08 00:51:02.273037] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46160][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326262272539000, start_id=1736326262272539000, end_id=1736326262272539001) [2025-01-08 00:51:02.282883] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=72] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:02.284948] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=39] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:02.285131] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=24] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326255686329000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:02.285377] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=29] gc stale ls task succ [2025-01-08 00:51:02.285436] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:02.285474] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=31] temporary file module use no blocks [2025-01-08 00:51:02.285487] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:02.285504] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:02.285521] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193766198373, flush_io_finished_ret_:0, flush_io_finished_round_:1925, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:02.286781] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84227957}], scn_array=[cnt:1, {val:1736326262272539000, v:0}], curr_size=1074, accum_size=2270, time_cost=13051) [2025-01-08 00:51:02.286960] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=53] Slow write(duration=12412, write_len=4096) [2025-01-08 00:51:02.286980] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=19] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8604, offset=2441226}, end_cursor:ObLogCursor{file_id=1, log_id=8605, offset=2441523}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894760050, len:4086, log_data_len:297, seq:8604, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:02.287366] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326262002109001, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:02.287503] INFO [SHARE] add_event (ob_event_history_table_operator.h:266) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1736326262287391), 'checkpoint', 'update_clog_checkpoint', 'tenant_id', 1002, 'ls_id', 1002, 'checkpoint_scn', '{val:1736326262002109001, v:0}', 'service_type', 'MAX_DECIDED_SCN', 'update_clog_checkpoint_times', 120, '', '192.168.159.137', 2882)) [2025-01-08 00:51:02.287640] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:02.287668] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=27] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:02.288569] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:02.288607] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=37] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:02.288657] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326256709695004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.288755] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=94] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:02.288776] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:02.288798] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:02.288814] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:02.288830] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2178335}, scn={val:1736326261717746002, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:02.291180] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8605, offset=2441523}, end_cursor:ObLogCursor{file_id=1, log_id=8606, offset=2441820}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894764050, len:3789, log_data_len:297, seq:8605, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:02.291351] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326261717746002, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:02.291382] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:02.291478] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=94] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:02.291492] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:02.295544] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=35] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:02.307844] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA4-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:02.307887] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA4-0-0] [lt=39] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:02.314690] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:3172K/788:181142 disk:3172K/788:181142 t1m2:52K/12:206354 t1m2g0:52K/12:206354 t1002m2:3044K/759:171324 t1002m2g0:3044K/759:171324 t1001m2:76K/17:600924 t1001m2g0:76K/17:600924 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:02.319209] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.319289] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.320653] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:02.325619] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] success to update min reserved snapshot(reserved_snapshot=1736324455686329000, duration=1800, snapshot_gc_ts=1736326255686329000) [2025-01-08 00:51:02.326474] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185419-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:02.330062] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=32] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324455686329000) [2025-01-08 00:51:02.330092] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=27] update reserved snapshot finished(cost_ts=4478, reserved_snapshot=1736324455686329000) [2025-01-08 00:51:02.360557] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.376512] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=41] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:02.377984] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=54] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:02.378128] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=42] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326255577817000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:02.378167] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=37] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326255577817000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326255577817000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:02.386509] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=17][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:02.390533] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:02.392697] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=24] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:02.394391] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=65] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:02.394446] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=39] temporary file module use no blocks [2025-01-08 00:51:02.394460] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:02.394474] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:02.394488] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193766308088, flush_io_finished_ret_:0, flush_io_finished_round_:1921, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:02.405028] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=66] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:02.405082] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=49] temporary file module use no blocks [2025-01-08 00:51:02.405095] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:02.405158] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=61] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:02.405171] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193766318712, flush_io_finished_ret_:0, flush_io_finished_round_:3749, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:02.416629] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6731165696, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:02.417387] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [50389][T1002_PLSSer][T1001][YB42C0A89F89-00062B2D24884C9A-0-0] [lt=43] need check schema(new_schema_version=1736326191428712, cached_tenant_schema_version=1736326191428712, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT * FROM __all_ls_status WHERE tenant_id = ? ORDER BY tenant_id, ls_id) [2025-01-08 00:51:02.417462] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=36] ====== traversal_flush timer task ====== [2025-01-08 00:51:02.417627] INFO [MDS] flush (mds_table_mgr.cpp:195) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84ED4-0-0] [lt=33] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326262331670000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269284252000, latest_log_ts:{val:1736326252703023001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.417728] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84ED4-0-0] [lt=95] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:02.422618] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=140] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:02.424330] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=154] gc stale ls task succ [2025-01-08 00:51:02.426852] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:02.428293] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=31] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:02.428325] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=27] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:51:02.428337] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:51:02.428348] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:02.431762] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=130, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}]) [2025-01-08 00:51:02.432112] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=345] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:02.432107] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:02.432132] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] try gc retain ctx [2025-01-08 00:51:02.432140] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:02.432170] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326261935253003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326251909938004, v:0}, keep_alive_scn_:{val:1736326256925355004, v:0}, update_ts_:1736326262427291}) [2025-01-08 00:51:02.432918] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=98] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326270087570000, latest_log_ts:{val:1736326253487654001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:02.433075] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:02.433116] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] finish update min start scn(max_decided_scn={val:1736326261935253003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326251909938004, v:0}, keep_alive_scn_:{val:1736326256925355003, v:0}, update_ts_:1736326262433107}) [2025-01-08 00:51:02.433843] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326261935253001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:02.434077] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=88] finish update min start scn(max_decided_scn={val:1736326262002109003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326254991003000, v:0}, keep_alive_scn_:{val:1736326260017888000, v:0}, update_ts_:1736326262433107}) [2025-01-08 00:51:02.435630] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=43] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326262002109001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:02.435729] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=89] finish update min start scn(max_decided_scn={val:1736326261717746002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326253203138002, v:0}, keep_alive_scn_:{val:1736326258210334002, v:0}, update_ts_:1736326262435643}) [2025-01-08 00:51:02.436437] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326261717746002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326259001815003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:02.440912] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:02.442471] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=152] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3351, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:02.442569] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=90] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=676, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:02.442633] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=26, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:02.445800] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=29][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:02.456487] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BEC5-0-0] [lt=27] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:02.458536] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.459404] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.460656] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.461329] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=50] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1736326261858850000, v:0}, base_version={val:1736326261858850000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326262262761000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326262461296, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326262331670000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326262541447, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326261858850000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326262120872, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:02.461437] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=106] no need to update min/max version(new_version={val:1736326261858850000, v:0}, min_version={val:1736326261858850000, v:0}, max_version={val:1736326261858850000, v:0}) [2025-01-08 00:51:02.484983] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-01-08 00:51:02.490467] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:02.507089] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=6247) [2025-01-08 00:51:02.518039] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F4AA-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326262517080000, start_id=1736326262517080000, end_id=1736326262517080001) [2025-01-08 00:51:02.518652] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD87-0-0] [lt=37] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326262608527, keepalive_ts:-1}) [2025-01-08 00:51:02.518684] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD87-0-0] [lt=30] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326262608527, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:02.519493] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.520106] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6F7-0-0] [lt=43] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=22, avg_log_submit_to_flush_time=4845, avg_log_submit_to_slide_time=7956) [2025-01-08 00:51:02.521238] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F4A8-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1736326262517080000, start_id=1736326262517080000, end_id=1736326262517080001) [2025-01-08 00:51:02.522759] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:02.526004] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB78-0-0] [lt=36][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193767439836, cache_obj->added_lc()=false, cache_obj->get_object_id()=13585, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:02.532344] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=142] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:02.542498] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.542791] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=64] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2972, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:02.542819] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9005) [2025-01-08 00:51:02.542838] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=187, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:02.542866] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=27, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:02.544405] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.545102] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38A8-0-0] [lt=50] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:02.545126] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38A8-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=5155, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:02.545137] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38A8-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=94, avg_log_submit_to_flush_time=3270, avg_log_submit_to_slide_time=5061) [2025-01-08 00:51:02.545300] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9005, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:02.560795] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.574333] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=43] PNIO [ratelimit] time: 1736326262574329, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007599, add_bytes: 0 [2025-01-08 00:51:02.575918] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5990) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=47] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000) [2025-01-08 00:51:02.575961] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.576142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.576304] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.576443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=80] get multi version start(ls_id={id:1}, tablet_id={id:233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.576569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=59] get multi version start(ls_id={id:1}, tablet_id={id:50114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.576632] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.576714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.576777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.576837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577001] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577093] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577174] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577292] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577357] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577444] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:101011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577623] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577685] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577745] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577805] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577867] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577927] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.577988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578078] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101098}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578282] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578350] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578652] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578742] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101036}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578830] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.578954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101064}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579067] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579242] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101080}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579390] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579515] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579685] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579756] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.579948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101083}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580266] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580334] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101062}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580457] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=61] get multi version start(ls_id={id:1}, tablet_id={id:60167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101018}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.580968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581286] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:101034}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581503] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581564] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581624] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101084}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581684] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581744] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581805] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581924] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.581984] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101068}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582398] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582591] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582765] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582826] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.582965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583247] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583318] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583488] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=59] get multi version start(ls_id={id:1}, tablet_id={id:60441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583704] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583784] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101073}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.583969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101099}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584150] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584288] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=81] get multi version start(ls_id={id:1}, tablet_id={id:392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584350] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101092}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584648] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.584962] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.585022] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.585082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101031}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.585169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.585310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.585497] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.585622] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.585807] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.585900] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.586041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.586108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.586169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.587500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=1268] get multi version start(ls_id={id:1}, tablet_id={id:60352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.587566] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.587623] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.587678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.587732] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.587787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.587858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:50292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.587919] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101067}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588266] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588334] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588516] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588628] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101027}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588767] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.588930] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589184] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589363] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589669] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589782] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101082}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589897] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.589976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590121] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590227] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590292] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590350] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590407] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590576] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590774] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590835] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.590950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591007] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591313] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101057}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591395] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:101001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591453] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591623] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591735] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591791] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591903] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.591959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101033}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592107] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592164] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592221] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=82] get multi version start(ls_id={id:1}, tablet_id={id:60293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592473] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101066}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592729] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592811] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592870] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592927] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.592983] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593040] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593153] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593279] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593420] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101030}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593497] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:101094}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101020}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593612] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101044}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593725] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:100002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593782] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593896] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.593952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594289] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594352] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594467] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594695] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101013}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.594961] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595253] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595319] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101072}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595350] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD3-0-0] [lt=25] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:02.595377] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595380] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD3-0-0] [lt=22] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=33) [2025-01-08 00:51:02.595456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595579] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595644] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101088}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595701] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595759] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101069}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595871] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595928] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.595984] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596193] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:60351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596440] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596496] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596553] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596610] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101060}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.596666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.597321] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=93] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:02.599075] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=95] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:02.596815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101100}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599382] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=2501] get multi version start(ls_id={id:1}, tablet_id={id:279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599506] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599564] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599622] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599736] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101022}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:50362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.599961] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600075] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600278] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600362] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600429] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600492] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600580] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.600950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.601009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.601069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.601128] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.601187] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.601917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101035}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:5}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602296] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602420] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101058}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602539] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602599] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602688] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101081}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101086}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101052}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.602953] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603012] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603072] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101019}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603190] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603306] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=63] get multi version start(ls_id={id:1}, tablet_id={id:101079}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603465] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:50272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603646] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603861] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.603985] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=68] get multi version start(ls_id={id:1}, tablet_id={id:406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604270] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604325] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:51:02.604344] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604465] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604761] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604853] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.604925] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605001] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605123] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605228] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605299] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605360] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605420] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605835] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.605895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606016] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606307] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606431] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606522] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606765] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606836] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.606925] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:50423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607037] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607267] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607407] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:50223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607798] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.607977] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608095] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608403] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608659] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.608992] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609053] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609112] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609171] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609316] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609390] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609451] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609511] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609570] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609869] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101043}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609933] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.609992] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610051] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610124] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610266] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610430] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610518] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:60121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:60151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610813] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.610986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:60496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.611053] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.611126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.611275] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.611343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.611886] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29A-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:02.611976] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29A-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326262002109004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326262002109004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:02.612757] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101032}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.612832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:101089}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.612892] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.612952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613012] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613109] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:50356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613237] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613391] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613455] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613515] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613574] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613632] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613692] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.613981] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614100] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614267] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614332] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614571] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614630] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614689] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614748] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614808] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614867] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101028}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.614986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615143] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615278] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615403] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615522] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101021}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.615958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616264] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616331] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616391] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616483] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616580] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616650] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.616885] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617007] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617071] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617352] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:50528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617421] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617481] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617600] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101037}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617660] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617779] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617868] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.617942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:101054}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618003] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618123] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618239] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:60433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618606] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618665] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101041}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618725] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618844] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.618904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101097}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619112] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619177] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619316] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=82] get multi version start(ls_id={id:1}, tablet_id={id:101024}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619413] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619536] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619869] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619928] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.619987] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620106] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101055}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=85] get multi version start(ls_id={id:1}, tablet_id={id:50400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620578] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620698] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620757] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.620995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621067] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621271] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621335] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101063}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621395] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621455] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621514] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621573] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:3}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621692] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621751] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.621828] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.622934] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:02.623585] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:02.623819] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.623893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.623885] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84230461}], scn_array=[cnt:1, {val:1736326262436684001, v:0}], curr_size=122, accum_size=3822, time_cost=17435) [2025-01-08 00:51:02.623955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101040}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624075] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624135] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624529] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624614] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:60153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101047}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:50265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.624971] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625117] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625240] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625313] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625444] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625531] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625612] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625733] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101059}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625852] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625912] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.625971] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626240] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626491] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626670] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626791] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.626924] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627268] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627330] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627653] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.627950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628246] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628431] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101046}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628490] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628594] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:60265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628721] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628840] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.628958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629078] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629267] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629333] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629366] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:02.629393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101045}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.629989] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630117] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630179] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630363] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=107] get multi version start(ls_id={id:1}, tablet_id={id:131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630458] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630547] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630625] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630685] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630743] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630915] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101078}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.630973] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631030] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631248] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101049}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631335] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631397] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631454] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631511] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631624] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101101}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631850] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.631897] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:02.631964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632284] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632458] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632503] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=107] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:02.632577] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632599] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:02.632619] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:02.632632] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:02.632668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101091}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632857] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.632986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101039}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633043] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633100] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101029}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633284] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:60495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633397] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633460] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633518] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633575] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.633687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.634301] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=143] Slow write(duration=18966, write_len=4096) [2025-01-08 00:51:02.634329] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=26] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8606, offset=2441820}, end_cursor:ObLogCursor{file_id=1, log_id=8607, offset=2442125}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894768050, len:3492, log_data_len:305, seq:8606, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:02.634423] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=52] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:02.634785] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58371][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=9] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:02.635174] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E47-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:02.635318] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E47-0-0] [lt=135] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326262436684000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326262436684000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:02.635567] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2189392}], scn_array=[cnt:1, {val:1736326262517080000, v:0}], curr_size=122, accum_size=732, time_cost=16276) [2025-01-08 00:51:02.635641] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=9] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:02.635970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.636060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.636155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.636272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.636561] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.636733] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1089) [2025-01-08 00:51:02.636815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.636879] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.636939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637051] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637117] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637177] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637382] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=119] get multi version start(ls_id={id:1}, tablet_id={id:402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:100004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637704] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637842] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90B6-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:02.637913] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90B6-0-0] [lt=69] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6888, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:02.637792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.637970] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90B6-0-0] [lt=55] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=410, avg_log_gen_to_submit_time=501, avg_log_submit_to_flush_time=4142, avg_log_submit_to_slide_time=6386) [2025-01-08 00:51:02.638077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=180] get multi version start(ls_id={id:1}, tablet_id={id:60500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.638141] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.638320] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.638385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.638770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101095}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.638863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:101070}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.638924] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.639005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.639070] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.639168] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.639333] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90B7-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=37, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:02.639423] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90B7-0-0] [lt=89] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6009, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:02.639603] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=107] get multi version start(ls_id={id:1}, tablet_id={id:254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.639674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101075}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.639734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.639793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.639864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.639944] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640006] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101077}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640064] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640122] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640246] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640307] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101071}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101093}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640641] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640694] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640796] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640870] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:101065}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.640987] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641294] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641605] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641746] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:100003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.641997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642117] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642177] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101014}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642331] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=97] get multi version start(ls_id={id:1}, tablet_id={id:60468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642449] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:50506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642861] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.642920] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643053] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=77] get multi version start(ls_id={id:1}, tablet_id={id:50103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643080] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:02.643116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643176] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643323] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=87] get multi version start(ls_id={id:1}, tablet_id={id:101004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643386] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643446] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643619] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643682] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643885] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.643947] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644034] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644301] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644362] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644519] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644764] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.644973] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.645095] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.645160] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.645330] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=111] get multi version start(ls_id={id:1}, tablet_id={id:60463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.645590] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=90] get multi version start(ls_id={id:1}, tablet_id={id:133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.645706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:101050}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.645807] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.645874] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.645978] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646073] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:100001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646271] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=73] get multi version start(ls_id={id:1}, tablet_id={id:243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646389] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646493] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:50327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101056}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.646945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647161] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647329] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=96] get multi version start(ls_id={id:1}, tablet_id={id:50244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=102] get multi version start(ls_id={id:1}, tablet_id={id:180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647580] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647782] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:101023}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.647951] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648102] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=90] get multi version start(ls_id={id:1}, tablet_id={id:146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648165] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=81] get multi version start(ls_id={id:1}, tablet_id={id:101048}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648403] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648494] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:101026}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648646] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648745] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.648994] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.649056] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.649124] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.649469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.649581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.649647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.649852] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.649955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.650018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.650939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651288] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651404] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:60518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:101025}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651629] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651688] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651744] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651799] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.651947] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:4}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652165] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:100005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652297] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652669] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652729] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652849] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.652970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653117] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653279] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653382] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101085}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653883] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.653973] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:101087}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654867] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.654954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655280] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:50427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:60315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655514] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655699] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:50501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655814] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:101051}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655867] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=10] get multi version start(ls_id={id:1}, tablet_id={id:100006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.655954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656014] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656067] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656153] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656282] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656340] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656424] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656518] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656584] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:101061}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656740] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656799] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.656914] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657001] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:101076}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657257] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657407] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657460] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:101006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657547] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657607] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657665] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.657991] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658052] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658105] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658286] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658390] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658517] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101074}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658574] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658664] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658754] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.658975] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659064] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101053}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659234] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659605] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659885] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60322}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.659941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.660029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.660091] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.660273] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=54] get multi version start(ls_id={id:1}, tablet_id={id:60107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.660365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.660428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.660488] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.660552] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.660641] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.661361] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.663127] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:02.663732] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AC2B-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.671243333816528320e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=383, Throttle Trigger(MB)=230, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=8.103636458226476407e+02) [2025-01-08 00:51:02.664008] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=62] submit log success(service_type_=0, last_id=1736326279284252000, limited_id=1736326279284252000) [2025-01-08 00:51:02.664094] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=83] submit log with lock success(service_type_=0, last_id=1736326279284252000, limited_id=1736326279284252000) [2025-01-08 00:51:02.664554] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50322}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.664651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.664716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.664817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.664913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.664976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=106] get multi version start(ls_id={id:1}, tablet_id={id:50102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665298] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=93] get multi version start(ls_id={id:1}, tablet_id={id:265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665389] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665451] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665768] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.665917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.666024] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.666120] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.666182] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.666367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=117] get multi version start(ls_id={id:1}, tablet_id={id:60208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.666431] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.666666] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43119071}], scn_array=[cnt:1, {val:1736326262436684002, v:0}], curr_size=122, accum_size=122, time_cost=16177) [2025-01-08 00:51:02.666531] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667353] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=728] get multi version start(ls_id={id:1}, tablet_id={id:400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667423] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667604] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:322}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667663] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667722] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101042}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667729] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B95-0-0] [lt=24] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:02.667781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667852] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.667934] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668000] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668177] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668340] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668505] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:60492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101096}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.668948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669032] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=55] get multi version start(ls_id={id:1}, tablet_id={id:60411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669258] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669354] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669418] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669503] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669736] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.669931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670160] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101038}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670425] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670627] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.670975] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=205] get multi version start(ls_id={id:1}, tablet_id={id:60238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.671064] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:101090}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.671162] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.671494] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=219] get multi version start(ls_id={id:1}, tablet_id={id:60328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.671605] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.671715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.671784] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324454278994000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324454278994000, max_merged_snapshot=1) [2025-01-08 00:51:02.671834] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:116) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E47-0-0] [lt=43] SSTableGCTask(cost_ts=96386) [2025-01-08 00:51:02.672380] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:02.672413] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=86) [2025-01-08 00:51:02.673856] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=21] update id meta(ret=0, service_type=0, limited_id=1736326279284252000, latest_scn={val:1736326262663505001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279284252000, latest_log_ts:{val:1736326262663505001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:02.673955] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=97] handle submit callback(service_type_=0, success=true, used_ts=9948, log_ts={val:1736326262663505001, v:0}, limited_id=1736326279284252000) [2025-01-08 00:51:02.673968] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=12] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:02.673976] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:02.673999] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] get number(ret=0, service_type_=0, range=1, base_id=1736326262673446000, start_id=1736326262673446000, end_id=1736326262673446001) [2025-01-08 00:51:02.674487] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326261929410000, v:0}, new_max_version={val:1736326261929410000, v:0}, last_min_version={val:1736326261858850000, v:0}, last_max_version={val:1736326261858850000, v:0}, total_time_us=12721, delta_us=745075) [2025-01-08 00:51:02.674536] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:02.684620] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84644-0-0] [lt=62] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2025-01-08 00:51:02.685937] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84644-0-0] [lt=50] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2025-01-08 00:51:02.685985] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1010) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84644-0-0] [lt=45] no transfer partition job [2025-01-08 00:51:02.686001] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1040) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84644-0-0] [lt=12] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer=true) [2025-01-08 00:51:02.689008] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:284) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84644-0-0] [lt=10] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-01-08 00:51:02.689074] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:493) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84644-0-0] [lt=53] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2025-01-08 00:51:02.689107] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:157) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84644-0-0] [lt=25] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=3, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1003}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone3", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1736324440080711, last_statistic_bg_stat_time=1736326192603087, last_statistic_schema_version=1736326192037600, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-01-08 00:51:02.689232] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585433-0-0] [lt=25] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:02.689282] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585433-0-0] [lt=41] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:02.692069] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=126] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5201059646, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:02.692231] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=43] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:02.692248] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5201059646, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:02.706611] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=52] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5783, avg_group_log_size=444) [2025-01-08 00:51:02.706653] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5783) [2025-01-08 00:51:02.706671] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:02.723702] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:02.723744] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:02.734128] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=118] ====== tenant freeze timer task ====== [2025-01-08 00:51:02.734102] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=173] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2616, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:02.734235] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=125] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=509, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:02.734567] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=49] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:02.761502] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.761781] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1736326262663505002, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43119854}, palf_committed_end_scn_:{val:1736326262663505002, v:0}, last_check_scn_:{val:1736326262663505002, v:0}, max_applied_cb_scn_:{val:1736326262663505002, v:0}}) [2025-01-08 00:51:02.761860] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=72] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43119854}, readable_log_tail:{lsn:43119854}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23984914, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:02.769171] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=59] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:02.783553] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=31] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:02.787666] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1001, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:17718, max_lsn:2156601, max_scn:{val:1736326262436684002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2156601}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2156479}, last_submit_end_lsn:{lsn:2156601}, last_submit_log_id:17718, last_submit_log_pid:1, max_flushed_lsn:{lsn:2156479}, max_flushed_end_lsn:{lsn:2156601}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2156601}, last_slide_log_id:17718, last_slide_scn:{val:1736326262436684002, v:0}, last_slide_lsn:{lsn:2156479}, last_slide_end_lsn:{lsn:2156601}, last_slide_log_pid:1, last_slide_log_accum_checksum:2251062219, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924f6a530}, [StateMgr]={this:0x2ad924f6e830, self:"192.168.159.137:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:17719, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324447115984, palf_role_change_cb:0x2ad924f752f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924f6bc70}, [ModeMgr]={palf_id:1001, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324447762546, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436756804001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/meta", dir_fd:177, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924f70a38, flashback_version:0}, log_storage:{log_tail:{lsn:2156601}, readable_log_tail:{lsn:2156601}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64948167, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1736324447114985, this:0x2ad924f6ef70}, [Reconfirm]={palf_id:1001, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:1"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324447117136, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924f6e9b0}) [2025-01-08 00:51:02.788259] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=584] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326262734389, last_flushed_end_lsn:{lsn:2156601}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326262737027, last_flushed_end_lsn:{lsn:2156601}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326261963081, last_flushed_end_lsn:{lsn:2155634}}]) [2025-01-08 00:51:02.788288] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25] current log_task status(palf_id=1001, self="192.168.159.137:2882", log_id=17719, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326262734389, last_flushed_end_lsn:{lsn:2156601}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326262737027, last_flushed_end_lsn:{lsn:2156601}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326261963081, last_flushed_end_lsn:{lsn:2155634}}]) [2025-01-08 00:51:02.801434] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D026-0-0] [lt=59][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:02.819265] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44ED-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:02.823034] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6FA-0-0] [lt=37] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:02.823065] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D6FA-0-0] [lt=30] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10306, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:02.823765] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6722777088, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:02.823930] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:02.826012] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:02.829517] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:02.834778] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=84] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:02.840010] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=51] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:02.840742] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] BlockGCTimerTask success(ret=0, cost_time_us=23, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:02.852247] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=27] ====== traversal_flush timer task ====== [2025-01-08 00:51:02.852554] INFO [MDS] flush (mds_table_mgr.cpp:195) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857A1-0-0] [lt=47] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326262436684003, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326269254996000, latest_log_ts:{val:1736326252704413001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:02.852722] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857A1-0-0] [lt=161] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:02.859705] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1003}, {id:1}]) [2025-01-08 00:51:02.861674] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.861882] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=203] no need to update min/max version(new_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}) [2025-01-08 00:51:02.888963] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=35] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:02.896999] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.897040] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.897350] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.897360] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=60] ====== check clog disk timer task ====== [2025-01-08 00:51:02.897385] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:02.897555] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:02.900620] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1736326262900616, bytes: 869642459, bw: 0.246317 MB/s, add_ts: 1000981, add_bytes: 258535 [2025-01-08 00:51:02.918751] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44EF-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:02.924086] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:02.924149] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1736326262436684003, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1736326262924116}, version_delta=487464) [2025-01-08 00:51:02.931119] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] ====== check clog disk timer task ====== [2025-01-08 00:51:02.931148] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:02.934849] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:02.938474] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=34] submit log success(service_type_=0, last_id=1736326279254996000, limited_id=1736326279254996000) [2025-01-08 00:51:02.938498] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=23] submit log with lock success(service_type_=0, last_id=1736326279254996000, limited_id=1736326279254996000) [2025-01-08 00:51:02.938535] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326262938370], gts:1736326262937841000, latest_srr:[mts=1736326262938370]}) [2025-01-08 00:51:02.938555] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:02.938581] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326262938566], gts:1736326262937841000, latest_srr:[mts=1736326262938566]}) [2025-01-08 00:51:02.938976] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326262938927], gts:1736326262937841000, latest_srr:[mts=1736326262938927]}) [2025-01-08 00:51:02.939630] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=133] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:02.941568] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=26] update id meta(ret=0, service_type=0, limited_id=1736326279254996000, latest_scn={val:1736326262937841001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279254996000, latest_log_ts:{val:1736326262937841001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:02.941757] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=187] handle submit callback(service_type_=0, success=true, used_ts=3283, log_ts={val:1736326262937841001, v:0}, limited_id=1736326279254996000) [2025-01-08 00:51:02.941799] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=40] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:02.941812] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=12] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:02.950880] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=4294, avg_group_log_size=357) [2025-01-08 00:51:02.950968] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=82] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4761) [2025-01-08 00:51:02.961749] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:02.962064] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] no need to update min/max version(new_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}) [2025-01-08 00:51:02.985354] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:02.990637] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:03.017563] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:03.018336] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58386][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=7] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:51:03.023534] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C8E-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:03.024451] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6722777088, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:03.025166] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C8E-0-0] [lt=88][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:03.026093] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:03.028122] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:03.028540] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:03.028982] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58351][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=51] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:51:03.029933] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:03.033637] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:03.035045] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:03.039495] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90BE-0-0] [lt=52] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:03.041029] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D34-0-0] [lt=45] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:03.041642] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=37] runTimer to refresh locality_info(ret=0) [2025-01-08 00:51:03.043735] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:346) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=18] set_server_locality_array success(server_locality_array=[{inited:true, addr:"192.168.159.137:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322580423131, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.138:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581270010, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.139:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581175973, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-01-08 00:51:03.043785] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=37] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-01-08 00:51:03.043796] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=11] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-01-08 00:51:03.043810] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=8] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1001, primary_zone=RANDOM) [2025-01-08 00:51:03.043817] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=7] add_locality_zone success(locality_zone={tenant_id:1001, region_priority:18446744073709551615}) [2025-01-08 00:51:03.043829] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1002, primary_zone=RANDOM) [2025-01-08 00:51:03.043836] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] add_locality_zone success(locality_zone={tenant_id:1002, region_priority:18446744073709551615}) [2025-01-08 00:51:03.043845] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1001, region_priority:18446744073709551615}, {tenant_id:1002, region_priority:18446744073709551615}]}) [2025-01-08 00:51:03.044317] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F85286-0-0] [lt=26] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326258146463, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:03.044415] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F85286-0-0] [lt=94] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326258148394, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:03.044437] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F85286-0-0] [lt=21] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, renew_time:1736326258150583, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, renew_time:1736326258150584, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, renew_time:1736326258150586, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326258150579, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:03.045525] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18659143}], scn_array=[cnt:1, {val:1736326262937841000, v:0}], curr_size=122, accum_size=1211, time_cost=11784) [2025-01-08 00:51:03.045564] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:03.045580] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2156845}], scn_array=[cnt:1, {val:1736326262937841000, v:0}], curr_size=122, accum_size=1089, time_cost=11826) [2025-01-08 00:51:03.061871] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.062148] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] no need to update min/max version(new_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}) [2025-01-08 00:51:03.070916] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D17285292-0-0] [lt=24] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:03.097572] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:03.099262] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=90] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:03.104506] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:51:03.108826] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4690, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:03.108857] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4438) [2025-01-08 00:51:03.108878] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=111, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:03.108911] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=56] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:03.110263] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67C3-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:03.110287] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67C3-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=6498, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:03.110298] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67C3-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=4219, avg_log_submit_to_slide_time=5829) [2025-01-08 00:51:03.110371] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4438, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:03.121931] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A44F3-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:03.123324] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4557, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:03.123412] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=341, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:03.123488] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=60, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:03.125899] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:03.125980] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:52447, [delay/us]:prepare:9, schedule:538, submit:346, rt:3160, total:4054) [2025-01-08 00:51:03.125999] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:03.126157] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=96] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52440, [delay/us]: prepare:9, schedule:538, submit:346, device:3160, total:4054) [2025-01-08 00:51:03.126222] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:03.126269] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48403, [delay/us]:prepare:7, schedule:153, submit:715, rt:2804, total:3680) [2025-01-08 00:51:03.126281] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:03.126321] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48403, [delay/us]: prepare:7, schedule:153, submit:715, device:2804, total:3680) [2025-01-08 00:51:03.126332] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:03.126354] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20168, [delay/us]:prepare:6, schedule:56, submit:1, rt:8717, total:8781) [2025-01-08 00:51:03.126418] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:31, bw:125043, [delay/us]:prepare:6, schedule:72, submit:536, rt:3180, total:3796) [2025-01-08 00:51:03.126430] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12100, [delay/us]:prepare:3, schedule:110, submit:899, rt:1888, total:2901) [2025-01-08 00:51:03.126440] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=8, iops=8, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:03.126471] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20168, [delay/us]: prepare:6, schedule:56, submit:1, device:8717, total:8781) [2025-01-08 00:51:03.126487] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12100, [delay/us]: prepare:3, schedule:110, submit:899, device:1888, total:2901) [2025-01-08 00:51:03.126494] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:31, avg_bw:125041, [delay/us]: prepare:6, schedule:72, submit:536, device:3180, total:3796) [2025-01-08 00:51:03.126504] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:03.126524] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4033, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:366, total:376) [2025-01-08 00:51:03.126548] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:03.126575] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4033, [delay/us]: prepare:0, schedule:0, submit:600000000, device:366, total:376) [2025-01-08 00:51:03.126593] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:03.126608] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:03.126616] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:03.126624] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:03.126632] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=144kB/s, net_out=170kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:03.127028] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=699] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:03.135255] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:03.135388] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=183] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:03.155646] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3870) [2025-01-08 00:51:03.162021] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.162606] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] no need to update min/max version(new_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}) [2025-01-08 00:51:03.163294] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:03.188864] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=41] ====== check clog disk timer task ====== [2025-01-08 00:51:03.188903] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:03.207412] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=93, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:03.224872] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6722777088, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:03.226772] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=57] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=566226, clean_num=31457) [2025-01-08 00:51:03.226925] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:51:03.231820] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:03.231873] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:03.233177] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87055-0-0] [lt=41][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:03.233316] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=109][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:03.233331] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:03.235623] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:03.235743] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:03.239049] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=105] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:03.251964] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=78, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:03.257557] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:202) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=7] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2025-01-08 00:51:03.257591] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=30] finish to resize log external storage handler(current_ts=1736326263257588, tenant_max_cpu=4, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:03.257606] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=13] snapshot_traffic_info once [2025-01-08 00:51:03.262215] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.262737] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] no need to update min/max version(new_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}) [2025-01-08 00:51:03.265385] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=22] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:03.265411] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:03.265437] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8497F-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:03.265463] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8497F-0-0] [lt=23] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:03.285552] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=19] blacklist_loop exec finished(cost_time=70, is_enabled=true, send_cnt=0) [2025-01-08 00:51:03.285722] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:03.285806] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:03.285834] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-01-08 00:51:03.285844] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:03.285858] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:03.285871] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193767199393, flush_io_finished_ret_:0, flush_io_finished_round_:1926, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:03.292105] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:03.292347] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=116][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%", tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:03.294844] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:03.294968] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%", tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-01-08 00:51:03.295725] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=55] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:03.310259] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA5-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:03.310298] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA5-0-0] [lt=36] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:03.319277] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=59] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.319337] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.321477] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:216K/52:188879 disk:216K/52:188879 t1m2:48K/11:587680 t1m2g0:48K/11:587680 t1002m2:124K/31:49970 t1002m2g0:124K/31:49970 t1001m2:44K/10:180816 t1001m2g0:44K/10:180816 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:03.325376] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17907, miss_cnt:0, cache_read_size:2190368, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:03.325716] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=44] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=92, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:03.325756] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=39] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:03.327293] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=78] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:03.329036] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918541A-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:03.334061] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1333, avg_group_log_size=121) [2025-01-08 00:51:03.334229] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1333) [2025-01-08 00:51:03.335599] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=46] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:03.336076] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=38] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:03.337016] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=98] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=22, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:03.337678] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=36] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:03.348744] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2025-01-08 00:51:03.362327] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:03.362374] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.363080] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A52-0-0] [lt=26] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:03.363322] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] no need to update min/max version(new_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}) [2025-01-08 00:51:03.394770] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=93] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:03.394821] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=44] temporary file module use no blocks [2025-01-08 00:51:03.394833] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:03.394847] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:03.394862] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193767308427, flush_io_finished_ret_:0, flush_io_finished_round_:1922, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:03.405455] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=121] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:03.405540] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=78] temporary file module use no blocks [2025-01-08 00:51:03.405553] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:03.405566] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:03.405580] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193767319138, flush_io_finished_ret_:0, flush_io_finished_round_:3750, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:03.425299] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=15] gc stale ls task succ [2025-01-08 00:51:03.429453] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:03.429868] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] ====== tenant freeze timer task ====== [2025-01-08 00:51:03.429898] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=69446440, tenant_id=1001) [2025-01-08 00:51:03.429920] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:03.433230] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=68] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6720679936, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:03.434164] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=69, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:03.436336] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:03.436577] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:03.439535] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=107] get number(ret=0, service_type_=0, range=1, base_id=1736326263438968000, start_id=1736326263438968000, end_id=1736326263438968001) [2025-01-08 00:51:03.440278] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46205][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=48] submit log success(service_type_=0, last_id=1736326280087570000, limited_id=1736326280087570000) [2025-01-08 00:51:03.440320] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46205][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=41] submit log with lock success(service_type_=0, last_id=1736326280087570000, limited_id=1736326280087570000) [2025-01-08 00:51:03.444541] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24] ====== tenant freeze timer task ====== [2025-01-08 00:51:03.448616] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] update id meta(ret=0, service_type=0, limited_id=1736326280087570000, latest_scn={val:1736326263438968001, v:0}, *this={tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326260017888004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326280087570000, latest_log_ts:{val:1736326263438968001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:03.448696] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=79] handle submit callback(service_type_=0, success=true, used_ts=8418, log_ts={val:1736326263438968001, v:0}, limited_id=1736326280087570000) [2025-01-08 00:51:03.448712] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=16] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:03.448721] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:03.452341] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=98] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:03.458574] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.460496] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.462467] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.463570] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] no need to update min/max version(new_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}) [2025-01-08 00:51:03.486835] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=103] clear dtl interm result cost(us)(clear_cost=7922, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:03.487536] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=68] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-01-08 00:51:03.492850] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=81] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:03.495638] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:03.495750] WDIAG read_multi_row (ob_table_access_helper.h:238) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:03.497362] INFO delete_row (ob_table_access_helper.h:339) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1001, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326263495768) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326263495768) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:03.509721] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1226) [2025-01-08 00:51:03.519574] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.521449] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F4AE-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326263521132000, start_id=1736326263521132000, end_id=1736326263521132001) [2025-01-08 00:51:03.526917] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D701-0-0] [lt=30] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=16, avg_log_submit_to_flush_time=4112, avg_log_submit_to_slide_time=6749) [2025-01-08 00:51:03.529616] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:03.529744] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:194) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=91] get wrs ts(ls_id={id:1}, delta=590551, log_service_ts=1736326262937841, min_tx_service_ts=4611686018427387, timestamp={val:1736326262937841005, v:0}) [2025-01-08 00:51:03.529769] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=48, Other_Error_Cnt=0, Submit_Succ_Cnt=52, last_scn="{val:1736326258720921001, v:0}", last_lsn={lsn:84213478}, last_gts={val:1736326263438968000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-01-08 00:51:03.536446] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] no need to update min/max version(new_version={val:1736326261513350002, v:0}, min_version={val:1736326261513350002, v:0}, max_version={val:1736326261513350002, v:0}) [2025-01-08 00:51:03.542152] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=152] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.547544] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D35-0-0] [lt=6] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:03.550415] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB7A-0-0] [lt=56][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193768464248, cache_obj->added_lc()=false, cache_obj->get_object_id()=13586, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:03.550553] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.554022] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=56] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4096, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:03.554092] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=68] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1759) [2025-01-08 00:51:03.554114] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=121, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:03.554147] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:03.556748] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38B2-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:03.556772] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38B2-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6206, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:03.556783] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38B2-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=287, avg_log_gen_to_submit_time=401, avg_log_submit_to_flush_time=4261, avg_log_submit_to_slide_time=5805) [2025-01-08 00:51:03.556905] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1759, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:03.562618] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.563709] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=49] no need to update min/max version(new_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}) [2025-01-08 00:51:03.577448] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8B-00062B2D183BE814-0-0] [lt=6][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1808, fly_ts=843527, send_timestamp=1736326262733917, peer="192.168.159.139:45082", sz=257) [2025-01-08 00:51:03.580774] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A5-0-0] [lt=51] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:03.580862] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A5-0-0] [lt=71] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=3, avg_log_batch_cnt=0, total_group_log_size=366, avg_group_log_size=122) [2025-01-08 00:51:03.580990] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A5-0-0] [lt=116] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:03.581434] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A5-0-0] [lt=420] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=11, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:03.581496] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A5-0-0] [lt=60] [PALF STAT LOG LIFE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=807234, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:03.581507] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A5-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=20, avg_log_submit_to_flush_time=3453, avg_log_submit_to_slide_time=429237) [2025-01-08 00:51:03.581521] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37A5-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17897, log_task={header:{begin_lsn:{lsn:2178335}, end_lsn:{lsn:2178457}, log_id:17897, min_scn:{val:1736326261717746003, v:0}, max_scn:{val:1736326261717746003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2178213}, prev_proposal_id:1, committed_end_lsn:{lsn:2178335}, data_checksum:940009244, accum_checksum:2205184695, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326262069001, freeze_ts:1736326262069001, submit_ts:1736326262069014, flushed_ts:1736326262072926, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:3912}, fs_cb_begin_ts=1736326263581419, log_life_time=1512418) [2025-01-08 00:51:03.583532] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=488, hit_cnt=4, read_cnt=4, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:03.583596] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17808, miss_cnt:0, cache_read_size:2178457, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:03.583699] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1736326263583696, bytes: 0, bw: 0.000000 MB/s, add_ts: 1009367, add_bytes: 0 [2025-01-08 00:51:03.585844] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2067, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:03.585887] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:03.585911] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=645, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:03.585977] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=61, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:03.587322] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46141][pnio1][T0][YB42C0A89F8B-00062B2D18EC32A9-0-0] [lt=30][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=1442348, send_timestamp=1736326262144971, peer="192.168.159.139:45058", sz=265) [2025-01-08 00:51:03.590981] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8461-0-0] [lt=30][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:2155634}, new_end_lsn:{lsn:2155756}, old_advance_time_us:1736326261963081, new_ack_time_us:1736326263590976, advance delay(us):1627895}) [2025-01-08 00:51:03.591406] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F89-00062B2D21F89B98-0-0] [lt=11][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5387, fly_ts=1287066, send_timestamp=1736326262304338, peer="192.168.159.139:45074", sz=480) [2025-01-08 00:51:03.595984] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F8B-00062B2D1CBBB5B5-0-0] [lt=61][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1308, fly_ts=1566177, send_timestamp=1736326262029805, peer="192.168.159.139:45062", sz=1632) [2025-01-08 00:51:03.597096] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58383][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF28-0-0] [lt=46] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326263593457, keepalive_ts:-1}) [2025-01-08 00:51:03.597121] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58383][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF28-0-0] [lt=24] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326263593457, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:03.597894] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:03.601818] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2AB-0-0] [lt=50] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326263592530, keepalive_ts:-1}) [2025-01-08 00:51:03.601864] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2AB-0-0] [lt=45] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326263592530, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:03.601980] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:03.604618] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=262] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:51:03.620477] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2AC-0-0] [lt=16] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326263592594, keepalive_ts:-1}) [2025-01-08 00:51:03.620510] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2AC-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326263592594, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:03.625063] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F0-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:03.625114] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F0-0-0] [lt=31] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326261717746003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326261717746003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:03.630408] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:03.631110] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58389][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:03.633740] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:03.634696] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=81] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:03.634728] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:03.634748] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:03.634763] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:03.636897] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6720679936, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:03.637157] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:03.645164] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F1-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326263593195, keepalive_ts:-1}) [2025-01-08 00:51:03.645247] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F1-0-0] [lt=81] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326263593195, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:03.649109] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3924, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:03.649132] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18659881}], scn_array=[cnt:1, {val:1736326263438968002, v:0}], curr_size=122, accum_size=738, time_cost=13006) [2025-01-08 00:51:03.649276] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=783, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:03.649403] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=26, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:03.649419] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8330, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:03.649428] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=123, avg_log_submit_to_flush_time=5020, avg_log_submit_to_slide_time=8206) [2025-01-08 00:51:03.649461] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:03.650118] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90CB-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:03.650391] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2157577}], scn_array=[cnt:1, {val:1736326263438968001, v:0}], curr_size=122, accum_size=610, time_cost=15416) [2025-01-08 00:51:03.650711] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=45] start to print blacklist info [2025-01-08 00:51:03.651157] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=37] ls blacklist refresh finish(cost_time=814823) [2025-01-08 00:51:03.651401] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:03.651443] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9137, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:03.651572] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43120830}], scn_array=[cnt:1, {val:1736326263637119000, v:0}], curr_size=588, accum_size=588, time_cost=12089) [2025-01-08 00:51:03.663305] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.663509] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:03.664749] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] no need to update min/max version(new_version={val:1736326261929410000, v:0}, min_version={val:1736326261929410000, v:0}, max_version={val:1736326261929410000, v:0}) [2025-01-08 00:51:03.681531] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] BlockGCTimerTask success(ret=0, cost_time_us=17, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:03.694422] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5202895387, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:03.694495] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:03.694511] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad908599450) [2025-01-08 00:51:03.694543] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5202895387, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:03.698524] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC38C-0-0] [lt=52][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:03.707962] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1226, avg_group_log_size=122) [2025-01-08 00:51:03.708012] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1226) [2025-01-08 00:51:03.708032] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:03.725676] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=155] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:03.729699] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=29] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:03.730531] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=110] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:03.732604] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857A3-0-0] [lt=25] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.787054533958435059e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=893, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.792507527813269519e+01) [2025-01-08 00:51:03.732934] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1736326263731517000, start_id=1736326263731517000, end_id=1736326263731517001) [2025-01-08 00:51:03.736443] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:03.738152] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=96] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:03.739105] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=18] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:03.740846] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=17131259, this={dio_aligned_buf:{buf_write_offset:2332, buf_padding_size:1764, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:70487, truncate_used_ts:17362}, log_block_size:67108864, io_fd:{first_id:-1, second_id:152, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=17129472, buf_len=545, total_write_size=44440, total_write_size_after_dio=507904, ob_pwrite_used_ts=338494, count=113, each_pwrite_cost=2995) [2025-01-08 00:51:03.741240] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4768, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:03.741389] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=128] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=78, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:03.749123] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:51:03.749313] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58393][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:03.749370] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58393][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67CA-0-0] [lt=23] init point thread id with(&point=0x5572d50e2f80, idx_=1049, point=[thread id=58393, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58393) [2025-01-08 00:51:03.749617] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=30] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:03.750445] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:03.751871] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708514B-0-0] [lt=25] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:03.752302] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=68][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18660003}], scn_array=[cnt:1, {val:1736326263438968003, v:0}], curr_size=122, accum_size=860, time_cost=13314) [2025-01-08 00:51:03.752466] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB APPEND COST TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=174, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:03.752513] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=46] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=6469, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:03.752526] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:03.752534] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=-728, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:03.752549] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=14] [APPLY STAT CB TOTAL TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=5914, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:03.756693] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC38F-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:03.764969] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=41] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326263438968001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=325991, in_cluster_service=true, cluster_version={val:1736326261929410000, v:0}, min_cluster_version={val:1736326261929410000, v:0}, max_cluster_version={val:1736326261929410000, v:0}, get_cluster_version_err=0, cluster_version_delta=1835549, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18643, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=609401742, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:03.768720] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDCC-0-0] [lt=57][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193768682552, cache_obj->added_lc()=false, cache_obj->get_object_id()=13587, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:03.769018] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.769446] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=41] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:03.771270] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=49] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326261942996999, v:0}, new_max_version={val:1736326261942996999, v:0}, last_min_version={val:1736326261929410000, v:0}, last_max_version={val:1736326261929410000, v:0}, total_time_us=6236, delta_us=1828271) [2025-01-08 00:51:03.771301] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326261942996999, v:0}, min_version={val:1736326261942996999, v:0}, max_version={val:1736326261942996999, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:03.771687] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:263) [50686][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1FF8458E-0-0] [lt=25] ObArchiveSender is running(thread_index=3) [2025-01-08 00:51:03.783904] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=117] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:03.796421] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D029-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:03.822215] INFO [LIB] stat (utility.h:1210) [58244][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4501-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:03.824845] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D707-0-0] [lt=163] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:03.824873] INFO [LIB] stat (utility.h:1210) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D707-0-0] [lt=28] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4590, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:03.826226] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:03.830663] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:03.836948] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326261717746004, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=2119191, in_cluster_service=true, cluster_version={val:1736326261513350003, v:0}, min_cluster_version={val:1736326261513350003, v:0}, max_cluster_version={val:1736326261513350003, v:0}, get_cluster_version_err=0, cluster_version_delta=2323587, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18208, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=491754464, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:03.837006] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=45] no need to update min/max version(new_version={val:1736326261513350003, v:0}, min_version={val:1736326261513350003, v:0}, max_version={val:1736326261513350003, v:0}) [2025-01-08 00:51:03.837376] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:03.837520] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:03.841426] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90D2-0-0] [lt=27] init point thread id with(&point=0x5572d50e2e80, idx_=1045, point=[thread id=58389, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58389) [2025-01-08 00:51:03.853403] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=36] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:03.853435] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=30] try gc retain ctx [2025-01-08 00:51:03.853444] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:03.853480] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326263763810000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326253696901001, v:0}, keep_alive_scn_:{val:1736326258743968001, v:0}, update_ts_:1736326263853000}) [2025-01-08 00:51:03.854447] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=78] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279284252000, latest_log_ts:{val:1736326262663505001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:03.859865] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1002}, {id:1}, {id:1001}]) [2025-01-08 00:51:03.868008] INFO [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1165) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=25] refresh queuing mode(ret=0, tenant_id=1001, stream_cnt=0, update_schema_cnt=0, cost_time=10) [2025-01-08 00:51:03.868040] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1215) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=30] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1001, interval_step=1, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1001, bucket_cnt:0, tenant_mem_user_limit:0, tenant_mem_user_hold:0, cached_node_array:0x2ad8f95d0ee0}) [2025-01-08 00:51:03.869247] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.894586] INFO [PALF] get_log (palf_handle_impl.cpp:4108) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2AD-0-0] [lt=9] submit fetch log task success(ret=0, palf_id=1, server="192.168.159.139:2882", fetch_type=0, prev_lsn={lsn:18657265}, start_lsn={lsn:18657810}, fetch_log_size=41943040, fetch_log_count=2048, accepted_mode_pid=1) [2025-01-08 00:51:03.894744] INFO [PALF] handle (fetch_log_engine.cpp:284) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=13] handle fetch_log_task(fetch_log_task={timestamp_us:1736326263894576, id:1, server:"192.168.159.139:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:1, prev_lsn:{lsn:18657265}, start_lsn:{lsn:18657810}, log_size:41943040, log_count:2048, accepted_mode_pid:1}) [2025-01-08 00:51:03.894862] INFO [PALF] next (palf_iterator.h:183) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=86] [PALF STAT ITERATOR INFO](io_ctx={user:"FETCHLOG", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:1, miss_cnt:0, cache_read_size:545, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:18657265}}}) [2025-01-08 00:51:03.895280] INFO [PALF] next (palf_iterator.h:183) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT ITERATOR INFO](io_ctx={user:"FETCHLOG", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:1, miss_cnt:0, cache_read_size:2437, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:18657810}}}) [2025-01-08 00:51:03.895341] WDIAG [PALF] batch_fetch_log_each_round_ (palf_handle_impl.cpp:3882) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4008] iterator next failed(ret=-4008, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:0x2ad9366f9000, next_round_pread_size:3678208, curr_read_pos:2437, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2437, log_storage_:{start_lsn:{lsn:18657810}, end_lsn:{lsn:18660247}, read_buf:{buf_len_:73728, buf_:0x2ad9366f9000}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18660247}, readable_log_tail:{lsn:18660247}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48444521, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326263438968004, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:2, group_size:66, proposal_id:1, committed_lsn:{lsn:18660125}, max_scn:{val:1736326263438968004, v:0}, accumulated_checksum:3678827773, log_id:19362, flag:20847}}, init_mode_version:0, accumulate_checksum:3678827773, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326263438968004, v:0}}, io_ctx:{user:"FETCHLOG", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:1, miss_cnt:0, cache_read_size:2437, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:18657810}}}}, replayable_point={val:1736326259001815003, v:0}, prev_lsn={lsn:18660125}) [2025-01-08 00:51:03.895504] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3819) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=61] fetch_log_from_storage_ finished(ret=0, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, prev_lsn={lsn:18657265}, iterator={iterator_impl:{buf_:0x2ad9366f9000, next_round_pread_size:3678208, curr_read_pos:2437, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2437, log_storage_:{start_lsn:{lsn:18657810}, end_lsn:{lsn:18660247}, read_buf:{buf_len_:73728, buf_:0x2ad9366f9000}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18660247}, readable_log_tail:{lsn:18660247}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48444521, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326263438968004, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:2, group_size:66, proposal_id:1, committed_lsn:{lsn:18660125}, max_scn:{val:1736326263438968004, v:0}, accumulated_checksum:3678827773, log_id:19362, flag:20847}}, init_mode_version:0, accumulate_checksum:3678827773, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326263438968004, v:0}}, io_ctx:{user:"FETCHLOG", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:1, miss_cnt:0, cache_read_size:2437, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:18657810}}}}, fetch_log_count=2048, has_fetched_log_count=20) [2025-01-08 00:51:03.895764] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=72] submit_change_mode_meta_req success(ret=0, palf_id=1, server="192.168.159.139:2882", mode_meta={version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_pid=1) [2025-01-08 00:51:03.895786] INFO [PALF] fetch_log_from_storage (palf_handle_impl.cpp:3601) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FETCH LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=83886080) [2025-01-08 00:51:03.895800] INFO [LIB] stat (utility.h:1210) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FETCH LOG TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=10991, this=0x2ad8f0ff8600, extra_info="") [2025-01-08 00:51:03.895811] INFO [LIB] stat (utility.h:1210) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FETCH LOG EXECUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=127708, this=0x2ad8f0ff8830, extra_info="") [2025-01-08 00:51:03.895819] INFO [PALF] handle (fetch_log_engine.cpp:321) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=7] handle fetch log task(ret=0, palf_id=1, handle_cost_time_us=1057, fetch_log_task={timestamp_us:1736326263894576, id:1, server:"192.168.159.139:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:1, prev_lsn:{lsn:18657265}, start_lsn:{lsn:18657810}, log_size:41943040, log_count:2048, accepted_mode_pid:1}, fetch_stat={total_size:2437, group_log_cnt:20, read_cost:0, get_cost:0, send_cost:95}) [2025-01-08 00:51:03.896952] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.897086] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32] LogLoopThread round_cost_time(us)(round_cost_time=23) [2025-01-08 00:51:03.897160] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.897392] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.897563] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:03.897110] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:19362, max_lsn:18660247, max_scn:{val:1736326263438968004, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:18660247}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:18660125}, last_submit_end_lsn:{lsn:18660247}, last_submit_log_id:19362, last_submit_log_pid:1, max_flushed_lsn:{lsn:18660125}, max_flushed_end_lsn:{lsn:18660247}, max_flushed_log_pid:1, committed_end_lsn:{lsn:18660247}, last_slide_log_id:19362, last_slide_scn:{val:1736326263438968004, v:0}, last_slide_lsn:{lsn:18660125}, last_slide_end_lsn:{lsn:18660247}, last_slide_log_pid:1, last_slide_log_accum_checksum:3678827773, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8fc454530}, [StateMgr]={this:0x2ad8fc458830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:19363, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324399554770, palf_role_change_cb:0x2ad8fc45f2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8fc455c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324399758968, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/meta", dir_fd:175, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8fc45aa38, flashback_version:0}, log_storage:{log_tail:{lsn:18660247}, readable_log_tail:{lsn:18660247}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48444521, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324399554279, this:0x2ad8fc458f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324399555093, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8fc4589b0}) [2025-01-08 00:51:03.897687] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=569] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326263836622, last_flushed_end_lsn:{lsn:18660247}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326263841481, last_flushed_end_lsn:{lsn:18660247}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326261951407, last_flushed_end_lsn:{lsn:18657932}}]) [2025-01-08 00:51:03.897777] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=86] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=19363, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326263836622, last_flushed_end_lsn:{lsn:18660247}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326263841481, last_flushed_end_lsn:{lsn:18660247}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326261951407, last_flushed_end_lsn:{lsn:18657932}}]) [2025-01-08 00:51:03.897821] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1002, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18000, max_lsn:2190978, max_scn:{val:1736326263518581002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2190978}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2190856}, last_submit_end_lsn:{lsn:2190978}, last_submit_log_id:18000, last_submit_log_pid:1, max_flushed_lsn:{lsn:2190856}, max_flushed_end_lsn:{lsn:2190978}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2190978}, last_slide_log_id:18000, last_slide_scn:{val:1736326263518581002, v:0}, last_slide_lsn:{lsn:2190856}, last_slide_end_lsn:{lsn:2190978}, last_slide_log_pid:1, last_slide_log_accum_checksum:3186479522, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad92435e530}, [StateMgr]={this:0x2ad924362830, self:"192.168.159.137:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.138:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18001, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad9243692f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326260180472, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326260180472, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad92435fc70}, [ModeMgr]={palf_id:1002, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436803442001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/meta", dir_fd:184, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924364a38, flashback_version:0}, log_storage:{log_tail:{lsn:2190978}, readable_log_tail:{lsn:2190978}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64913790, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:-1, this:0x2ad924362f70}, [Reconfirm]={palf_id:1002, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9243629b0}) [2025-01-08 00:51:03.898128] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=300][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:03.898151] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:03.898163] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:03.898173] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] current log_task status(palf_id=1002, self="192.168.159.137:2882", log_id=18001, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:51:03.900736] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=51] PNIO [ratelimit] time: 1736326263900733, bytes: 869910676, bw: 0.255762 MB/s, add_ts: 1000117, add_bytes: 268217 [2025-01-08 00:51:03.900882] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58244][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8486-0-0] [lt=38][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:18657932}, new_end_lsn:{lsn:18658054}, old_advance_time_us:1736326261951407, new_ack_time_us:1736326263900879, advance delay(us):1949472}) [2025-01-08 00:51:03.908111] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:03.908143] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=30] try gc retain ctx [2025-01-08 00:51:03.908151] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:03.908177] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] get_max_applied_scn(scn={val:1736326263731517000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84232476}, palf_committed_end_scn_:{val:1736326263731517000, v:0}, last_check_scn_:{val:1736326263731517000, v:0}, max_applied_cb_scn_:{val:1736326263731517000, v:0}}) [2025-01-08 00:51:03.908290] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=107] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84232476}, readable_log_tail:{lsn:84232476}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49977060, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:03.908373] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=80] finish update min start scn(max_decided_scn={val:1736326263731517000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326253770280001, v:0}, keep_alive_scn_:{val:1736326258720921001, v:0}, update_ts_:1736326263907374}) [2025-01-08 00:51:03.909292] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279254996000, latest_log_ts:{val:1736326262937841001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:03.913341] INFO [RPC] process (ob_batch_processor.cpp:106) [58306][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=25] batch rpc statistics(clog_batch_nodelay_cnt=0, clog_batch_cnt=1, trx_batch_cnt=0, sql_batch_cnt=0) [2025-01-08 00:51:03.921678] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58387][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2C0D-0-0] [lt=47][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:43116938}, new_end_lsn:{lsn:43118238}, old_advance_time_us:1736326262491362, new_ack_time_us:1736326263921675, advance delay(us):1430313}) [2025-01-08 00:51:03.925836] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B1D-0-0] [lt=25][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:84226053}, new_end_lsn:{lsn:84226175}, old_advance_time_us:1736326261825333, new_ack_time_us:1736326263925834, advance delay(us):2100501}) [2025-01-08 00:51:03.928903] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=46] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8607, offset=2442125}, end_cursor:ObLogCursor{file_id=1, log_id=8608, offset=2442422}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89476c050, len:3187, log_data_len:297, seq:8607, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:03.930806] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:03.932013] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4503-0-0] [lt=107] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:03.932631] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E47-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:03.932688] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E47-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326263438968003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326263438968003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:03.936544] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F0-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326263935370000, start_id=1736326263935370001, end_id=1736326263935370002) [2025-01-08 00:51:03.940828] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:03.941150] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=520, get_gts_with_stc_cnt=3, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=3, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:03.942407] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:03.944324] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=112] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4208, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:03.951270] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C29A-0-0] [lt=83] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=6118, avg_group_log_size=407) [2025-01-08 00:51:03.951318] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C29A-0-0] [lt=43] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5490) [2025-01-08 00:51:03.955013] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=37] LogLoopThread round_cost_time(us)(round_cost_time=15) [2025-01-08 00:51:03.956807] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=84] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8608, offset=2442422}, end_cursor:ObLogCursor{file_id=1, log_id=8609, offset=2442719}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894770050, len:2890, log_data_len:297, seq:8608, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:03.959359] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=48] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8609, offset=2442719}, end_cursor:ObLogCursor{file_id=1, log_id=8610, offset=2443016}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894774050, len:2593, log_data_len:297, seq:8609, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:03.969374] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:03.982542] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=45] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:03.987777] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-01-08 00:51:03.993050] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=73] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:04.014550] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=72] ====== checkpoint timer task ====== [2025-01-08 00:51:04.015503] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=31] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:04.015551] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=48] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:04.015659] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279284252000, latest_log_ts:{val:1736326262663505001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:04.015773] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=111] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:04.015894] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:04.015916] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=19] get rec log scn(service_type_=0, rec_log_ts={val:1736326262663505001, v:0}) [2025-01-08 00:51:04.016033] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=115] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:04.016093] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=57] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:04.016109] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=8] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:04.016128] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:04.016142] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:04.016163] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:04.016258] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=92] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:04.016281] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=20] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:04.025037] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=18] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:04.031533] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] no need to update min/max version(new_version={val:1736326261820694000, v:0}, min_version={val:1736326261820694000, v:0}, max_version={val:1736326261820694000, v:0}) [2025-01-08 00:51:04.033793] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:04.035506] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58373][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=56] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:51:04.035790] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:04.037115] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C8F-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:04.038368] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=115] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:04.038550] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:04.038898] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:04.040117] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C8F-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:04.041319] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326264038888000, start_id=1736326264038888000, end_id=1736326264038888001) [2025-01-08 00:51:04.041410] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90D7-0-0] [lt=24] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1348) [2025-01-08 00:51:04.052591] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D36-0-0] [lt=6] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:04.069593] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:04.069654] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:04.092114] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D197BB4CE-0-0] [lt=162][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:51:04.098313] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:04.102468] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:04.105481] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:04.110873] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2378, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:04.110923] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1634) [2025-01-08 00:51:04.110948] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=944, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:04.110985] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=58] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=71, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:04.117149] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67CD-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:04.117338] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67CD-0-0] [lt=186] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7253, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:04.117356] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67CD-0-0] [lt=16] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=127, avg_log_submit_to_flush_time=3355, avg_log_submit_to_slide_time=7125) [2025-01-08 00:51:04.117471] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1634, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:04.124475] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4507-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:04.126010] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1893, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:04.126086] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=74] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:04.126111] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=128, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:04.126420] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=74, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:04.135488] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:04.135545] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:10, bw:40570, [delay/us]:prepare:11, schedule:97, submit:247, rt:1856, total:2213) [2025-01-08 00:51:04.135560] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:04.135614] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:10, avg_bw:40576, [delay/us]: prepare:11, schedule:97, submit:247, device:1856, total:2213) [2025-01-08 00:51:04.135628] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:04.135668] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:73041, [delay/us]:prepare:18, schedule:271, submit:847, rt:1885, total:3023) [2025-01-08 00:51:04.135678] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:04.135738] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:73040, [delay/us]: prepare:18, schedule:271, submit:847, device:1885, total:3023) [2025-01-08 00:51:04.135751] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:04.135774] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12173, [delay/us]:prepare:24, schedule:40, submit:1, rt:4034, total:4100) [2025-01-08 00:51:04.135797] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:35, bw:142022, [delay/us]:prepare:16, schedule:158, submit:445, rt:2551, total:3172) [2025-01-08 00:51:04.135837] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=8, iops=8, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:04.135870] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:3, avg_bw:12173, [delay/us]: prepare:24, schedule:40, submit:1, device:4034, total:4100) [2025-01-08 00:51:04.135942] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:35, avg_bw:142018, [delay/us]: prepare:16, schedule:158, submit:445, device:2551, total:3172) [2025-01-08 00:51:04.135958] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:04.136035] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4057, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:392, total:400) [2025-01-08 00:51:04.136064] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:04.136147] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=77] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4057, [delay/us]: prepare:0, schedule:0, submit:600000000, device:392, total:400) [2025-01-08 00:51:04.136167] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:04.136237] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:04.136246] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:04.136254] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:04.136262] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=320kB/s, net_out=230kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:04.138879] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB7C-0-0] [lt=57][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193769052669, cache_obj->added_lc()=false, cache_obj->get_object_id()=13588, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:04.141562] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1348) [2025-01-08 00:51:04.141988] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:04.155624] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58394][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:04.157368] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58395][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:04.157436] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:51:04.159253] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB81-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1736326264158683000, start_id=1736326264158683000, end_id=1736326264158683001) [2025-01-08 00:51:04.162840] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58395][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67CF-0-0] [lt=44] init point thread id with(&point=0x5572d50e3000, idx_=1051, point=[thread id=58395, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58395) [2025-01-08 00:51:04.163112] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=10] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:04.163782] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:04.169430] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=7720) [2025-01-08 00:51:04.169893] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:04.170262] INFO [SQL.ENG] reset_leader_statistics (ob_px_tenant_target_monitor.cpp:388) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D186BE709-0-0] [lt=59] reset leader statistics(tenant_id_=1, ret=0, version_=328951380616979, server_index=1) [2025-01-08 00:51:04.170303] INFO [SQL.ENG] process (ob_px_rpc_processor.cpp:629) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D186BE709-0-0] [lt=39] need refresh all(tenant_id=1, arg_.need_refresh_all_=true, follower_version=328950041959395, prev_leader_server_index=1, leader_server_index=1) [2025-01-08 00:51:04.187644] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:04.190119] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=57] get number(ret=0, service_type_=0, range=1, base_id=1736326264189426000, start_id=1736326264189426000, end_id=1736326264189426001) [2025-01-08 00:51:04.227432] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=20] gc stale ls task succ [2025-01-08 00:51:04.231452] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:04.231484] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:04.236086] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8705B-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:04.236132] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:04.236144] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:04.238886] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=74] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:04.239449] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1163872, replace_num=15728) [2025-01-08 00:51:04.239492] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:04.242473] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:04.270001] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:04.282994] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread began to shrink(tenant_->id()=1002, group_id_=19, token=0) [2025-01-08 00:51:04.285837] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=20] gc stale ls task succ [2025-01-08 00:51:04.286065] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:04.286130] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=56] temporary file module use no blocks [2025-01-08 00:51:04.286146] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:04.286160] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:04.286174] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193768199743, flush_io_finished_ret_:0, flush_io_finished_round_:1927, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:04.293090] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58135][T1002_L0_G19][T1002][Y0-0000000000000000-0-0] [lt=80] worker thread exit(tenant_->id()=1002, workers_.get_size()=1) [2025-01-08 00:51:04.295913] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=33] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:04.297750] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF47-0-0] [lt=26] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326264395613, keepalive_ts:-1}) [2025-01-08 00:51:04.297783] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF47-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326264395613, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:04.308822] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=72] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=85, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:04.313316] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA6-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:04.313361] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA6-0-0] [lt=42] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:04.319260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.319288] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.324710] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=113] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:288K/68:156218 disk:288K/68:156218 t1m2:44K/11:131618 t1m2g0:44K/11:131618 t1002m2:164K/39:133600 t1002m2g0:164K/39:133600 t1001m2:80K/18:217537 t1001m2g0:80K/18:217537 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:04.327479] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=204] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17917, miss_cnt:0, cache_read_size:2191588, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:04.327743] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=192, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:04.327768] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:04.329360] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=106] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:04.329463] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=97] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:04.329481] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=16] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:04.329490] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=8] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:04.332448] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=5] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:04.333510] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=1058] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:04.335406] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918541B-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:04.339805] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1649, avg_group_log_size=164) [2025-01-08 00:51:04.339834] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1649) [2025-01-08 00:51:04.370230] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:04.395156] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:04.395208] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=44] temporary file module use no blocks [2025-01-08 00:51:04.395488] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=276] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:04.395510] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:04.395540] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193768308735, flush_io_finished_ret_:0, flush_io_finished_round_:1923, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:04.399466] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=69] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326260180472, last_first_register_time_us=1736324449183670, register_time_us=1736326260180472, curr_time_us=1736326264399463) [2025-01-08 00:51:04.399720] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=124] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326264399593, last_submit_register_req_time_us=1736326264399593, register_parent_reason=1) [2025-01-08 00:51:04.399741] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:04.424891] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=32] report RowHolderMapper summary info(count=1, bkt_cnt=248) [2025-01-08 00:51:04.424848] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=35] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=3, inc_cnt=1, queue_size=3) [2025-01-08 00:51:04.425331] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [58396][IO_SYNC_CH3][T0][Y0-0000000000000000-0-0] [lt=0] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:4, threads_idle_time:5204728547, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:04.425664] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=47] gc stale ls task succ [2025-01-08 00:51:04.425977] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [58396][IO_SYNC_CH3][T0][Y0-0000000000000000-0-0] [lt=36] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:5, threads_idle_time:5204728548, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:04.629439] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.629661] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:04.629726] INFO pn_ratelimit (group.c:653) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=37] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:04.629883] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.629965] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=96] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:04.630053] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:04.630177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=170][errcode=0] task scheduled too delay(scheduled_delay=323711, expected_run_ts_us_=1736326264306463, ObClockGenerator::getRealClock()=1736326264630175, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:04.306, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:04.630605] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50380][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=222938, expected_run_ts_us_=1736326264407663, ObClockGenerator::getRealClock()=1736326264630603, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:1, time_interval:100.00ms, expected_run_time:00:51:04.407, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:04.630793] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=18] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=660295680, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=413446144, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:04.630876] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=39] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:04.630896] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2814, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:04.630964] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:04.631025] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:04.631038] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:04.632059] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:04.634130] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.634167] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=143] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.634190] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.636187] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=50] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:04.638048] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:04.638728] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58360][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=32] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:04.639331] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2191588}], scn_array=[cnt:1, {val:1736326264021522003, v:0}], curr_size=122, accum_size=366, time_cost=214651) [2025-01-08 00:51:04.639427] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84234106}], scn_array=[cnt:1, {val:1736326264337992000, v:0}], curr_size=585, accum_size=1858, time_cost=297623) [2025-01-08 00:51:04.641469] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=21618, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:04.641516] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43129859}], scn_array=[cnt:1, {val:1736326264341511000, v:0}], curr_size=589, accum_size=9618, time_cost=298917) [2025-01-08 00:51:04.641546] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=309, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:04.641573] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=52710, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:04.641587] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2179180}], scn_array=[cnt:1, {val:1736326264109018001, v:0}], curr_size=122, accum_size=723, time_cost=303865) [2025-01-08 00:51:04.641840] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=248] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=1557, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:04.641878] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=97][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2158431}], scn_array=[cnt:1, {val:1736326264189426001, v:0}], curr_size=122, accum_size=244, time_cost=301807) [2025-01-08 00:51:04.641942] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=105][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18661158}], scn_array=[cnt:1, {val:1736326264189426001, v:0}], curr_size=122, accum_size=667, time_cost=301896) [2025-01-08 00:51:04.642430] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=51] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=32, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:04.642494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=209036, expected_run_ts_us_=1736326264433455, ObClockGenerator::getRealClock()=1736326264642492, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:04.433, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:04.642595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=68][errcode=0] task scheduled too delay(scheduled_delay=190295, expected_run_ts_us_=1736326264452299, ObClockGenerator::getRealClock()=1736326264642594, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:04.452, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:04.642625] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=153044, expected_run_ts_us_=1736326264489581, ObClockGenerator::getRealClock()=1736326264642625, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff570, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:04.489, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:04.642646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=144900, expected_run_ts_us_=1736326264497745, ObClockGenerator::getRealClock()=1736326264642645, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:04.497, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:04.642663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=137563, expected_run_ts_us_=1736326264505100, ObClockGenerator::getRealClock()=1736326264642663, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfc950, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfcc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:04.505, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:04.642680] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=139752, expected_run_ts_us_=1736326264502927, ObClockGenerator::getRealClock()=1736326264642679, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:04.502, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:04.643946] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=377] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=10398720) [2025-01-08 00:51:04.644144] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1736326264644141, bytes: 0, bw: 0.000000 MB/s, add_ts: 1060445, add_bytes: 0 [2025-01-08 00:51:04.644364] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=49] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:04.645889] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:04.645924] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4015] clock out of order(cur_ts=1736326264645921, cur_ts_=1736326264428511, delta=217410) [2025-01-08 00:51:04.646059] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=39, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:04.646083] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:04.646969] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90E0-0-0] [lt=36][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=17734, log_task={header:{begin_lsn:{lsn:2158431}, end_lsn:{lsn:2158553}, log_id:17734, min_scn:{val:1736326264189426001, v:0}, max_scn:{val:1736326264189426001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2158431}, data_checksum:2222604892, accum_checksum:1973327310, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326264339960, freeze_ts:1736326264339964, submit_ts:1736326264340002, flushed_ts:1736326264646077, gen_to_freeze cost time:4, gen_to_submit cost time:42, submit_to_flush cost time:306075}, fs_cb_begin_ts=1736326264646955, log_life_time=306995) [2025-01-08 00:51:04.647353] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8B-00062B2D1D7BF127-0-0] [lt=52][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=167328, send_timestamp=1736326264480022, peer="192.168.159.139:45082", sz=265) [2025-01-08 00:51:04.650449] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46141][pnio1][T0][YB42C0A89F8B-00062B2D1BBBF295-0-0] [lt=36][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=148770, send_timestamp=1736326264501677, peer="192.168.159.139:45058", sz=265) [2025-01-08 00:51:04.651645] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58395][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B39-0-0] [lt=59][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=44527, log_task={header:{begin_lsn:{lsn:84234106}, end_lsn:{lsn:84234691}, log_id:44527, min_scn:{val:1736326264337992000, v:0}, max_scn:{val:1736326264337992000, v:0}, data_len:529, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84234106}, data_checksum:705002851, accum_checksum:3707467217, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326264338612, freeze_ts:1736326264338620, submit_ts:1736326264338727, flushed_ts:1736326264640339, gen_to_freeze cost time:8, gen_to_submit cost time:115, submit_to_flush cost time:301612}, fs_cb_begin_ts=1736326264651632, log_life_time=313020) [2025-01-08 00:51:04.651751] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=20][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326264337992000, v:0}, log_ts:{val:1736326264337992000, v:0}, lsn:{lsn:84234162}, submit_ts:1736326264338596}, this:0x2ad89c0ac2d8, is_inited_:true, trans_id:{txid:27885}, ls_id:{id:1}, ctx:0x2ad89c0a88d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=313150, ctx_lock_wait_time=1) [2025-01-08 00:51:04.652806] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90DE-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:04.652828] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90DE-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=52117, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:04.652838] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90DE-0-0] [lt=8][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=19367, log_task={header:{begin_lsn:{lsn:18661158}, end_lsn:{lsn:18661280}, log_id:19367, min_scn:{val:1736326264189426001, v:0}, max_scn:{val:1736326264189426001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18661158}, data_checksum:667309872, accum_checksum:2489608730, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326264339729, freeze_ts:1736326264339733, submit_ts:1736326264339795, flushed_ts:1736326264646094, gen_to_freeze cost time:4, gen_to_submit cost time:66, submit_to_flush cost time:306299}, fs_cb_begin_ts=1736326264652793, log_life_time=313064) [2025-01-08 00:51:04.652918] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90DF-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:04.652929] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90DF-0-0] [lt=11] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=65045, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:04.652937] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90DF-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17904, log_task={header:{begin_lsn:{lsn:2179180}, end_lsn:{lsn:2179302}, log_id:17904, min_scn:{val:1736326264109018001, v:0}, max_scn:{val:1736326264109018001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2179058}, prev_proposal_id:1, committed_end_lsn:{lsn:2179180}, data_checksum:940009244, accum_checksum:4126069793, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326264337559, freeze_ts:1736326264337559, submit_ts:1736326264337581, flushed_ts:1736326264646051, gen_to_freeze cost time:0, gen_to_submit cost time:22, submit_to_flush cost time:308470}, fs_cb_begin_ts=1736326264652906, log_life_time=315347) [2025-01-08 00:51:04.655048] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58398][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:04.655107] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE27-0-0] [lt=26] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:04.655130] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE27-0-0] [lt=15] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:04.655157] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE27-0-0] [lt=25] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:04.659568] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46389][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB82-0-0] [lt=22][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193769573401, cache_obj->added_lc()=false, cache_obj->get_object_id()=13589, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:04.660145] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=18] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:04.660424] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38C2-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=21, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:04.660432] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F8B-00062B2D18EC32C0-0-0] [lt=34][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=163536, send_timestamp=1736326264496893, peer="192.168.159.139:45074", sz=265) [2025-01-08 00:51:04.660450] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38C2-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=25462, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:04.660461] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38C2-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=202, avg_log_gen_to_submit_time=305, avg_log_submit_to_flush_time=22040, avg_log_submit_to_slide_time=25157) [2025-01-08 00:51:04.660474] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38C2-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=32479, log_task={header:{begin_lsn:{lsn:43129859}, end_lsn:{lsn:43130448}, log_id:32479, min_scn:{val:1736326264341511000, v:0}, max_scn:{val:1736326264341511000, v:0}, data_len:533, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43129859}, data_checksum:1223055976, accum_checksum:1767303549, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326264341678, freeze_ts:1736326264341682, submit_ts:1736326264341732, flushed_ts:1736326264642347, gen_to_freeze cost time:4, gen_to_submit cost time:54, submit_to_flush cost time:300615}, fs_cb_begin_ts=1736326264660378, log_life_time=318700) [2025-01-08 00:51:04.660704] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D710-0-0] [lt=41] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=24, avg_log_submit_to_flush_time=26352, avg_log_submit_to_slide_time=39571) [2025-01-08 00:51:04.660731] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58381][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D710-0-0] [lt=26][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18006, log_task={header:{begin_lsn:{lsn:2191588}, end_lsn:{lsn:2191710}, log_id:18006, min_scn:{val:1736326264021522003, v:0}, max_scn:{val:1736326264021522003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2191466}, prev_proposal_id:1, committed_end_lsn:{lsn:2191588}, data_checksum:1715074492, accum_checksum:1424367418, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326264424368, freeze_ts:1736326264424368, submit_ts:1736326264424391, flushed_ts:1736326264646017, gen_to_freeze cost time:0, gen_to_submit cost time:23, submit_to_flush cost time:221626}, fs_cb_begin_ts=1736326264660677, log_life_time=236309) [2025-01-08 00:51:04.662146] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326264341511000, v:0}, log_ts:{val:1736326264341511000, v:0}, lsn:{lsn:43129915}, submit_ts:1736326264341673}, this:0x2ad8fb6c0b58, is_inited_:true, trans_id:{txid:66653}, ls_id:{id:1}, ctx:0x2ad8fb6bd150, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=320469, ctx_lock_wait_time=1) [2025-01-08 00:51:04.662442] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9618, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:04.662512] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=41] succ to submit log task to replay service(to_submit_lsn={lsn:2191644}, to_submit_scn={val:1736326264021522003, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326264660679, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2191710}, next_to_submit_scn_:{val:1736326264021522004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2191588}, end_lsn:{lsn:2191710}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2191710}, readable_log_tail:{lsn:2191710}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64913058, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326264021522003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326264021522003, v:0}, data_checksum:3348837161, flag:61}}, init_mode_version:0, accumulate_checksum:1424367418, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326264021522003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17918, miss_cnt:0, cache_read_size:2191710, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:04.663961] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:04.665380] INFO [SERVER.OMT] recv_group_request (ob_tenant.cpp:1413) [46142][pnio1][T0][YB42C0A89F8A-00062B2D1E69BE27-0-0] [lt=41] worker thread created(id()=1002, group->group_id_=19) [2025-01-08 00:51:04.667976] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=99][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326264369973000, v:0}, log_ts:{val:1736326264369973000, v:0}, lsn:{lsn:43130504}, submit_ts:1736326264370933}, this:0x2ad8fb6fffd8, is_inited_:true, trans_id:{txid:66654}, ls_id:{id:1}, ctx:0x2ad8fb6fc5d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=297028, ctx_lock_wait_time=12) [2025-01-08 00:51:04.668512] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58395][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67D1-0-0] [lt=37] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3126) [2025-01-08 00:51:04.724385] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC392-0-0] [lt=61][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:04.728923] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=31] ====== tenant freeze timer task ====== [2025-01-08 00:51:04.729429] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=112] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=85, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:04.729510] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=32, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:04.729530] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=8, avg_log_batch_cnt=1, total_group_log_size=976, avg_group_log_size=122) [2025-01-08 00:51:04.729548] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=976) [2025-01-08 00:51:04.729562] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:04.729748] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=82] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:04.731331] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10329) [2025-01-08 00:51:04.732110] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:04.732322] WDIAG read_multi_row (ob_table_access_helper.h:238) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:04.734275] INFO delete_row (ob_table_access_helper.h:339) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=18] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326264732342) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326264732342) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:04.734539] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=2882, avg_group_log_size=320) [2025-01-08 00:51:04.734564] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2882) [2025-01-08 00:51:04.734578] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:04.734591] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:04.736236] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:219) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:04.736265] INFO [COMMON] handle (memory_dump.cpp:538) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:04.737726] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] update_check_range(min_check_version=372, max_check_version=373, global_version=374) [2025-01-08 00:51:04.738482] INFO [LIB] stat (utility.h:1210) [58389][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37B3-0-0] [lt=50] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:04.738552] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58389][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37B3-0-0] [lt=56] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=845, avg_group_log_size=120) [2025-01-08 00:51:04.738572] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58389][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37B3-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=845) [2025-01-08 00:51:04.742565] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90E3-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=976) [2025-01-08 00:51:04.748046] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=104] [APPLY STAT CB APPEND COST TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=140, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:04.748074] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=27] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=13803, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:04.748083] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:04.748091] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=-1107, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:04.748098] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=12837, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:04.749415] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84234691}], scn_array=[cnt:1, {val:1736326264428511000, v:0}], curr_size=122, accum_size=1980, time_cost=14524) [2025-01-08 00:51:04.749616] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=47658, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:04.749656] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2158553}], scn_array=[cnt:1, {val:1736326264428511000, v:0}], curr_size=122, accum_size=366, time_cost=15239) [2025-01-08 00:51:04.749695] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=798, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:04.750316] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:04.750340] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=47606, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:04.753882] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690DDB-0-0] [lt=28] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=78) [2025-01-08 00:51:04.753923] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690DDB-0-0] [lt=24] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.791854529380798340e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=895, Throttle Trigger(MB)=537, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.762787788122533783e+01) [2025-01-08 00:51:04.754265] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326264749586000, start_id=1736326264749586000, end_id=1736326264749586001) [2025-01-08 00:51:04.759298] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=44613, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:04.759492] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=191][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18661280}], scn_array=[cnt:1, {val:1736326264428511000, v:0}], curr_size=122, accum_size=789, time_cost=24951) [2025-01-08 00:51:04.759613] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=113] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=1292, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:04.759659] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2179302}], scn_array=[cnt:1, {val:1736326264428511001, v:0}], curr_size=122, accum_size=845, time_cost=21006) [2025-01-08 00:51:04.759732] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=845) [2025-01-08 00:51:04.759833] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=16, avg_log_gen_to_submit_time=90, avg_log_submit_to_flush_time=46725, avg_log_submit_to_slide_time=49319) [2025-01-08 00:51:04.759874] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:04.760817] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2191710}], scn_array=[cnt:1, {val:1736326264665540000, v:0}], curr_size=122, accum_size=488, time_cost=13134) [2025-01-08 00:51:04.772477] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:04.772767] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=73] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326264158683000, v:0}, new_max_version={val:1736326264158683000, v:0}, last_min_version={val:1736326263731517001, v:0}, last_max_version={val:1736326263731517001, v:0}, total_time_us=20611, delta_us=614082) [2025-01-08 00:51:04.772870] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=98] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326264158683000, v:0}, min_version={val:1736326264158683000, v:0}, max_version={val:1736326264158683000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:04.772895] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:04.777780] INFO handle (memory_dump.cpp:665) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 151 261 0 1 2 68 1 0 1 4 1 1 0 1 5 43 5 0 1 7 1 2 0 1 8 48 1 0 1 12 7 3 0 1 13 1 1 0 1 14 1 2 0 1 15 1 1 0 1 18 4 3 0 1 20 1 2 0 500 0 94 237 0 500 7 7 4 0 500 8 36 1 0 500 9 2 1 0 500 10 5 2 0 500 16 2 1 0 500 17 9 7 0 500 22 79 62 0 500 23 23 10 0 508 0 2 10 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 104 230 0 1001 2 49 1 0 1001 4 1 1 0 1001 5 22 5 0 1001 8 30 1 0 1001 12 3 2 0 1001 13 1 1 0 1001 14 1 1 0 1001 18 4 3 0 1001 20 2 2 0 1002 0 162 253 0 1002 2 59 1 0 1002 4 2 1 0 1002 5 49 6 0 1002 7 11 2 0 1002 8 46 1 0 1002 12 2 3 0 1002 13 1 1 0 1002 14 1 1 0 1002 18 9 3 0 1002 20 1 2 0 cost_time: 40141 [2025-01-08 00:51:04.777840] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=36] [MEMORY] tenant: 1, limit: 2,147,483,648 hold: 1,040,691,200 cache_hold: 213,909,504 cache_used: 213,909,504 cache_item_count: 102 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 445,685,760 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 142,606,336 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 95,862,784 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 100,663,296 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 14,680,064 limit= 429,496,720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:04.778323] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 445,685,760 used= 341,762,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,211,520 used= 31,196,160 count= 15 avg_used= 2,079,744 block_cnt= 15 chunk_cnt= 15 mod=CACHE_MAP_NODE [MEMORY] hold= 31,096,832 used= 31,033,536 count= 8 avg_used= 3,879,192 block_cnt= 8 chunk_cnt= 7 mod=IoControl [MEMORY] hold= 24,518,656 used= 24,371,776 count= 23 avg_used= 1,059,642 block_cnt= 23 chunk_cnt= 17 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,196,912 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 37 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,268,096 used= 11,228,160 count= 4 avg_used= 2,807,040 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,618,560 used= 7,541,760 count= 10 avg_used= 754,176 block_cnt= 10 chunk_cnt= 8 mod=ResourceGroup [MEMORY] hold= 7,151,616 used= 6,144,000 count= 123 avg_used= 49,951 block_cnt= 123 chunk_cnt= 23 mod=[T]char [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMemCU [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,391,488 used= 2,971,968 count= 138 avg_used= 21,536 block_cnt= 138 chunk_cnt= 28 mod=DiagnosticInfo [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,768,896 used= 2,720,000 count= 13 avg_used= 209,230 block_cnt= 13 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 2,751,360 used= 2,635,488 count= 580 avg_used= 4,543 block_cnt= 564 chunk_cnt= 29 mod=WorkerMap [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,640 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 272 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,394,368 used= 1,372,072 count= 75 avg_used= 18,294 block_cnt= 75 chunk_cnt= 11 mod=TabletMap [MEMORY] hold= 1,374,720 used= 1,308,288 count= 36 avg_used= 36,341 block_cnt= 36 chunk_cnt= 14 mod=CommSysVarFac [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,179,648 used= 1,175,808 count= 8 avg_used= 146,976 block_cnt= 8 chunk_cnt= 7 mod=SQLSessionInfo [MEMORY] hold= 960,192 used= 936,448 count= 102 avg_used= 9,180 block_cnt= 102 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 917,504 used= 450,688 count= 56 avg_used= 8,048 block_cnt= 56 chunk_cnt= 19 mod=[T]ObPerfEventR [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 506,752 used= 486,208 count= 107 avg_used= 4,544 block_cnt= 107 chunk_cnt= 25 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 262,144 used= 150,272 count= 16 avg_used= 9,392 block_cnt= 16 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 262,144 used= 261,120 count= 4 avg_used= 65,280 block_cnt= 4 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 203,840 used= 194,432 count= 49 avg_used= 3,968 block_cnt= 49 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 13 mod=LinearHashMapNo [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 11 mod=SqlSession [MEMORY] hold= 155,024 used= 2,177 count= 1,920 avg_used= 1 block_cnt= 19 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 154,432 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 13 mod=QueryAllocator [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,456 used= 131,072 count= 2 avg_used= 65,536 block_cnt= 2 chunk_cnt= 2 mod=MinorMergeMgr [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 105,664 used= 103,168 count= 13 avg_used= 7,936 block_cnt= 13 chunk_cnt= 9 mod=SqlSessiVarMap [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 7 mod=PlanVaIdx [MEMORY] hold= 93,328 used= 24,224 count= 343 avg_used= 70 block_cnt= 79 chunk_cnt= 26 mod=Coro [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 5 mod=HashNodeLCSta [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 54,720 used= 51,840 count= 15 avg_used= 3,456 block_cnt= 12 chunk_cnt= 11 mod=CACHE_INST [MEMORY] hold= 51,872 used= 40,320 count= 60 avg_used= 672 block_cnt= 59 chunk_cnt= 20 mod=[T]ObWarningBuf [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 40,400 used= 28,224 count= 63 avg_used= 448 block_cnt= 33 chunk_cnt= 11 mod=OMT_Worker [MEMORY] hold= 37,984 used= 4,032 count= 168 avg_used= 24 block_cnt= 68 chunk_cnt= 27 mod=[T]MemoryContex [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PartCtx [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,640 used= 15,872 count= 4 avg_used= 3,968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnMemNode [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,640 used= 6,320 count= 12 avg_used= 526 block_cnt= 12 chunk_cnt= 8 mod=SqlSessiQuerSql [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=SqlSessionSbloc [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 7,504 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 6 mod=PackStateMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 6 mod=SequenceMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 5 mod=SequenceIdMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 7 mod=ContextsMap [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,848 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 5 mod=DICollector [MEMORY] hold= 4,864 used= 4,288 count= 3 avg_used= 1,429 block_cnt= 2 chunk_cnt= 2 mod=NegParamIdex [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 2 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 320 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSIDDLVar [MEMORY] hold= 341,762,944 used= 334,167,794 count= 31,597 avg_used= 10,575 mod=SUMMARY [2025-01-08 00:51:04.778409] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=68] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 142,606,336 used= 141,492,224 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 block_cnt= 68 chunk_cnt= 68 mod=Memstore [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:04.778432] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:04.778452] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 95,862,784 used= 89,614,144 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83,164,416 used= 75,169,382 count= 2,878 avg_used= 26,118 block_cnt= 2,878 chunk_cnt= 43 mod=SqlPhyPlan [MEMORY] hold= 4,684,672 used= 4,444,848 count= 310 avg_used= 14,338 block_cnt= 310 chunk_cnt= 40 mod=SqlPlanCache [MEMORY] hold= 1,340,416 used= 1,222,144 count= 616 avg_used= 1,984 block_cnt= 206 chunk_cnt= 39 mod=CreateContext [MEMORY] hold= 422,656 used= 412,672 count= 52 avg_used= 7,936 block_cnt= 52 chunk_cnt= 12 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 89,614,144 used= 81,250,646 count= 3,858 avg_used= 21,060 mod=SUMMARY [2025-01-08 00:51:04.778470] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,368 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,272 used= 58,751 count= 259 avg_used= 226 block_cnt= 21 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,368 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-01-08 00:51:04.778483] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 100,663,296 used= 97,542,144 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97,542,144 used= 97,360,704 count= 189 avg_used= 515,136 block_cnt= 189 chunk_cnt= 48 mod=CoStack [MEMORY] hold= 97,542,144 used= 97,360,704 count= 189 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:04.778502] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 213,909,504 used= 213,909,504 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:04.778517] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 14,680,064 used= 12,713,920 limit= 429,496,720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,036,352 used= 7,149,419 count= 109 avg_used= 65,591 block_cnt= 109 chunk_cnt= 5 mod=L_TabletPool [MEMORY] hold= 4,587,520 used= 4,569,600 count= 70 avg_used= 65,280 block_cnt= 70 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 12,713,920 used= 11,803,787 count= 181 avg_used= 65,214 mod=SUMMARY [2025-01-08 00:51:04.778530] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,584,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 block_cnt= 195 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:04.778544] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 16,256 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ExtInfoLog [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:04.778556] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 33,216 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 mod=SUMMARY [2025-01-08 00:51:04.778578] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 14,001,280 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 9,473,984 used= 9,445,184 count= 3 avg_used= 3,148,394 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 4,519,168 used= 4,412,416 count= 556 avg_used= 7,936 block_cnt= 556 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 14,001,280 used= 13,865,536 count= 560 avg_used= 24,759 mod=SUMMARY [2025-01-08 00:51:04.778621] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=26] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 73,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 73,152 used= 71,424 count= 9 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:04.778661] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 631,246,848 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 232,894,464 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 24,260,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 170,676,224 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98,557,952 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:04.779049] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 232,894,464 used= 185,989,024 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,397,376 used= 7,340,256 count= 7 avg_used= 1,048,608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7,259,504 used= 6,324,350 count= 1,032 avg_used= 6,128 block_cnt= 540 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,783,552 used= 5,752,960 count= 6 avg_used= 958,826 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 16 mod=Iteratorget_ls_id()={id:1}) [2025-01-08 00:51:04.897654] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:04.897910] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=17] ====== check clog disk timer task ====== [2025-01-08 00:51:04.897933] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=18] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:04.898186] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.898302] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.898332] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.898353] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:04.898622] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=93] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:04.907926] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=276] PNIO [ratelimit] time: 1736326264907923, bytes: 870139437, bw: 0.216606 MB/s, add_ts: 1007190, add_bytes: 228761 [2025-01-08 00:51:04.918126] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=65] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:04.932580] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=19] ====== check clog disk timer task ====== [2025-01-08 00:51:04.932642] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=56] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:04.936731] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:04.944692] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4513-0-0] [lt=66] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=976) [2025-01-08 00:51:04.953411] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:04.953504] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1736326264749586000, v:0}, base_version={val:1736326264158683000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326264749586000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326264953487, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326264749586000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326264935868, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326264749586000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326265040819, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:04.953801] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=42] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12812226560, fast_delete_log_mode=false) [2025-01-08 00:51:04.955936] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F1-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:04.956003] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F1-0-0] [lt=49] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326264428511002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326264428511002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:04.958107] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=41] [APPLY STAT CB APPEND COST TIME](cur_stat_count=46, stat_interval=5000000, avg (count/cost)=160, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:04.958134] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=26] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=46, stat_interval=5000000, avg (count/cost)=5440, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:04.958260] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=46, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:04.958277] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=16] [APPLY STAT CB EXECUTE TIME](cur_stat_count=46, stat_interval=5000000, avg (count/cost)=-1338, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:04.958284] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=46, stat_interval=5000000, avg (count/cost)=4262, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:04.967627] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F1-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=6993, avg_group_log_size=537) [2025-01-08 00:51:04.967669] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F1-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=6993) [2025-01-08 00:51:04.971826] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:05.030725] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:05.031528] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=108] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:05.031819] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:32487, max_lsn:43134387, max_scn:{val:1736326264970363000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:43134387}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:1191347}}, last_submit_lsn:{lsn:43133798}, last_submit_end_lsn:{lsn:43134387}, last_submit_log_id:32487, last_submit_log_pid:1, max_flushed_lsn:{lsn:43133798}, max_flushed_end_lsn:{lsn:43134387}, max_flushed_log_pid:1, committed_end_lsn:{lsn:43134387}, last_slide_log_id:32487, last_slide_scn:{val:1736326264970363000, v:0}, last_slide_lsn:{lsn:43133798}, last_slide_end_lsn:{lsn:43134387}, last_slide_log_pid:1, last_slide_log_accum_checksum:3914142588, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad9083ee530}, [StateMgr]={this:0x2ad9083f2830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:32488, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324348477669, palf_role_change_cb:0x2ad9083f92f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad9083efc70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324348678844, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/meta", dir_fd:172, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad9083f4a38, flashback_version:0}, log_storage:{log_tail:{lsn:43134387}, readable_log_tail:{lsn:43134387}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23970381, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324348477523, this:0x2ad9083f2f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.138:2882";"192.168.159.139:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324348478003, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9083f29b0}) [2025-01-08 00:51:05.032164] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=339] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326264973970, last_flushed_end_lsn:{lsn:43134387}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326264975285, last_flushed_end_lsn:{lsn:43134387}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326264975895, last_flushed_end_lsn:{lsn:43134387}}]) [2025-01-08 00:51:05.032257] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=88] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=32488, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326264973970, last_flushed_end_lsn:{lsn:43134387}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326264975285, last_flushed_end_lsn:{lsn:43134387}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326264975895, last_flushed_end_lsn:{lsn:43134387}}]) [2025-01-08 00:51:05.034933] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1399, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:05.037317] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.037738] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B95-0-0] [lt=13] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:05.044709] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:05.049371] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:195) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C90-0-0] [lt=54][errcode=-4018] table_operator_ get_source failed(ret=-4018) [2025-01-08 00:51:05.049376] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58299][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=13] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:05.050808] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C90-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:05.052865] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C90-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:05.129270] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21] ts response task statistics(alloc_count=59754, free_count=29876) [2025-01-08 00:51:05.129371] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=90] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:05.132655] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=65] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:05.136501] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:05.138072] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.142147] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB84-0-0] [lt=28][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193770055981, cache_obj->added_lc()=false, cache_obj->get_object_id()=13590, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:05.144668] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:172) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=9] Eliminate task evict sql audit(request_manager_->get_tenant_id()=1001, queue_size=524288, config_mem_limit_=32212254, request_manager_->get_size_used()=0, evict_high_size_level=471859, evict_low_size_level=419430, allocator->allocated()=14558208, evict_high_mem_level=16106127, evict_low_mem_level=0) [2025-01-08 00:51:05.144693] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:05.144870] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:05.146110] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:05.147609] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29B-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:05.147654] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29B-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326264665540003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326264665540003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:05.147836] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37BB-0-0] [lt=78][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17908, log_task={header:{begin_lsn:{lsn:2179668}, end_lsn:{lsn:2179790}, log_id:17908, min_scn:{val:1736326264428511004, v:0}, max_scn:{val:1736326264428511004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2179546}, prev_proposal_id:1, committed_end_lsn:{lsn:2179668}, data_checksum:940009244, accum_checksum:451580287, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326265036705, freeze_ts:1736326265036705, submit_ts:1736326265036712, flushed_ts:1736326265039838, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:3126}, fs_cb_begin_ts=1736326265147825, log_life_time=111120) [2025-01-08 00:51:05.148243] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4517-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=8, avg_log_batch_cnt=0, total_group_log_size=967, avg_group_log_size=120) [2025-01-08 00:51:05.151069] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84B5-0-0] [lt=87] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1277) [2025-01-08 00:51:05.156809] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:05.157107] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=12] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:05.157172] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=58] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:05.157186] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=11] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:05.157207] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=19] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:05.159755] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=48] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8610, offset=2443016}, end_cursor:ObLogCursor{file_id=1, log_id=8611, offset=2443321}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894778050, len:2296, log_data_len:305, seq:8610, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:05.160764] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=43] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:05.160838] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43134387}], scn_array=[cnt:1, {val:1736326265128357000, v:0}], curr_size=122, accum_size=3350, time_cost=29077) [2025-01-08 00:51:05.162446] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E48-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:05.162457] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18661768}], scn_array=[cnt:1, {val:1736326265128357000, v:0}], curr_size=122, accum_size=1277, time_cost=25839) [2025-01-08 00:51:05.162485] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1277) [2025-01-08 00:51:05.162485] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E48-0-0] [lt=21] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326264428511003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326264428511003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:05.162616] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2159041}], scn_array=[cnt:1, {val:1736326265128357000, v:0}], curr_size=122, accum_size=854, time_cost=25290) [2025-01-08 00:51:05.162645] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=854) [2025-01-08 00:51:05.162805] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2179790}], scn_array=[cnt:1, {val:1736326264428511005, v:0}], curr_size=122, accum_size=488, time_cost=13287) [2025-01-08 00:51:05.162926] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58395][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=26] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:51:05.163196] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=56] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=30960, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:05.163393] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=195][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2192198}], scn_array=[cnt:1, {val:1736326264665540004, v:0}], curr_size=113, accum_size=967, time_cost=12824) [2025-01-08 00:51:05.163413] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=967) [2025-01-08 00:51:05.163445] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=810, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:05.164448] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=96] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:05.166556] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=202] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:05.167965] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=20] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:05.167984] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=18] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:05.169452] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084877-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:05.169515] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084877-0-0] [lt=18] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:05.169529] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084877-0-0] [lt=11] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:05.172436] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=12] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:05.173529] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=45, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:05.174122] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=38649, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:05.174150] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84236107}], scn_array=[cnt:1, {val:1736326265128357000, v:0}], curr_size=122, accum_size=3396, time_cost=13787) [2025-01-08 00:51:05.174223] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=71] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3396) [2025-01-08 00:51:05.174256] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2850, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:05.174495] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:05.174530] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=44488, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:05.174542] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=82, avg_log_submit_to_flush_time=42640, avg_log_submit_to_slide_time=44406) [2025-01-08 00:51:05.174565] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=41, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:05.174597] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3396, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:05.174631] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=24] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:05.174831] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=6569) [2025-01-08 00:51:05.175983] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=18] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:05.178527] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8611, offset=2443321}, end_cursor:ObLogCursor{file_id=1, log_id=8612, offset=2443618}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89477c050, len:1991, log_data_len:297, seq:8611, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:05.178807] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=255] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:05.178825] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=18] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:05.179056] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E48-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:05.179091] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E48-0-0] [lt=17] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326264428511003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326264428511003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:05.182824] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084877-0-0] [lt=7] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:05.183533] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084877-0-0] [lt=656] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:05.183582] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084877-0-0] [lt=46] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:05.189410] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] ====== check clog disk timer task ====== [2025-01-08 00:51:05.189457] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=38] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:05.190471] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=23] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8612, offset=2443618}, end_cursor:ObLogCursor{file_id=1, log_id=8613, offset=2443915}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894780050, len:1694, log_data_len:297, seq:8612, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:05.191388] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=32] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:05.192264] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=158] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8613, offset=2443915}, end_cursor:ObLogCursor{file_id=1, log_id=8614, offset=2444212}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894784050, len:1397, log_data_len:297, seq:8613, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:05.206133] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=115] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:05.207659] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=104] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:05.207706] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=45] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:05.209475] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084877-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:05.209576] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084877-0-0] [lt=47] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:05.209618] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084877-0-0] [lt=38] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:05.209645] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=16] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:05.210293] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=644] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:05.210348] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=46] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:05.210360] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:05.210381] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084877-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:05.210399] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084877-0-0] [lt=18] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:05.210409] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=6] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:05.210414] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=4] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:05.210429] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084877-0-0] [lt=13] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:05.210464] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084877-0-0] [lt=34] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:05.210480] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=10] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:05.210487] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:05.210513] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084877-0-0] [lt=23] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:05.211831] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084877-0-0] [lt=1316] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:05.213284] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=11] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:05.213328] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=40] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:05.213349] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=19] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:05.213386] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=9] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:05.213399] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:05.213408] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=9] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:05.213420] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:05.213436] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084877-0-0] [lt=15] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:05.227525] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=35] gc stale ls task succ [2025-01-08 00:51:05.230826] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=93] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:05.232043] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735339520, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:05.237849] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=19] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:05.238240] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.239366] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=117] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:05.239456] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326264299814000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:05.239483] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326264299814000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326264299814000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:05.242150] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:05.242176] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:05.242532] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:05.242579] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:9, bw:40699, [delay/us]:prepare:9, schedule:870, submit:8474, rt:25901, total:35255) [2025-01-08 00:51:05.242593] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:05.242646] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:40699, [delay/us]: prepare:9, schedule:870, submit:8474, device:25901, total:35255) [2025-01-08 00:51:05.242659] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:05.242698] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:16, bw:73999, [delay/us]:prepare:9, schedule:550, submit:4885, rt:14702, total:20148) [2025-01-08 00:51:05.242708] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:05.242788] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:17, avg_bw:74001, [delay/us]: prepare:9, schedule:550, submit:4885, device:14702, total:20148) [2025-01-08 00:51:05.242804] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:05.242828] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:14799, [delay/us]:prepare:7, schedule:16, submit:1, rt:2902, total:2927) [2025-01-08 00:51:05.242851] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4592, iops:30, bw:136896, [delay/us]:prepare:6, schedule:249, submit:6596, rt:32542, total:39394) [2025-01-08 00:51:05.242861] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=8, iops=8, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:05.242892] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:14800, [delay/us]: prepare:7, schedule:16, submit:1, device:2902, total:2927) [2025-01-08 00:51:05.242908] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4592, avg_iops:30, avg_bw:136907, [delay/us]: prepare:6, schedule:249, submit:6596, device:32542, total:39394) [2025-01-08 00:51:05.242918] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:05.242937] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3700, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:789, total:797) [2025-01-08 00:51:05.242960] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:05.242986] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3700, [delay/us]: prepare:0, schedule:0, submit:600000000, device:789, total:797) [2025-01-08 00:51:05.243003] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:05.243069] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:05.243079] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:05.243087] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:05.243094] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SERVER] net_in=189kB/s, net_out=189kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:05.255363] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=93] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=854, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:05.256828] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87061-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:05.256896] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:05.256911] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:05.257522] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2192311}], scn_array=[cnt:1, {val:1736326265146284000, v:0}], curr_size=122, accum_size=122, time_cost=12615) [2025-01-08 00:51:05.257691] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=26] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:05.257792] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=97] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=308, cache_node_num=308) [2025-01-08 00:51:05.257803] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=308, cache_node_num=308) [2025-01-08 00:51:05.257813] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=8] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:05.260349] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=308, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}]) [2025-01-08 00:51:05.261088] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=734] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:05.266526] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=42] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:05.266546] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=19] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:05.266595] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E6-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:05.266609] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E6-0-0] [lt=12] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:05.272905] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get_max_applied_scn(scn={val:1736326265128357000, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2159285}, palf_committed_end_scn_:{val:1736326265128357001, v:0}, last_check_scn_:{val:1736326265128357001, v:0}, max_applied_cb_scn_:{val:1736326265128357000, v:0}}) [2025-01-08 00:51:05.272942] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2159285}, readable_log_tail:{lsn:2159285}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64945483, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:05.282471] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854CC-0-0] [lt=24] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:05.282504] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854CC-0-0] [lt=26] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:05.285994] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=47] gc stale ls task succ [2025-01-08 00:51:05.291354] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285965-0-0] [lt=18] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12274, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:05.297253] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:05.298647] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:05.298738] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326255686329000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:05.298764] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=24] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326255686329000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326255686329000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:05.299319] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:05.300009] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285965-0-0] [lt=25] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12274, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265299983, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265299983, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265299983, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:05.300083] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285965-0-0] [lt=71] success to process renew task(task={tenant_id:1, table_id:12274, add_timestamp:1736326265290362}, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265299983, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265299983, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265299983, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:05.315415] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA7-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:05.315517] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA7-0-0] [lt=98] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:05.319352] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.319390] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.335604] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF48-0-0] [lt=26] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326265432885, keepalive_ts:-1}) [2025-01-08 00:51:05.335651] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF48-0-0] [lt=45] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326265432885, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:05.337151] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=100] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:236K/54:463111 disk:236K/54:463111 t1m2:32K/7:1089191 t1m2g0:32K/7:1089191 t1002m2:140K/33:234114 t1002m2g0:140K/33:234114 t1001m2:64K/14:689051 t1001m2g0:64K/14:689051 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:05.337635] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=377, gc_interval=10000000) [2025-01-08 00:51:05.337683] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=26] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:05.337703] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] gc_check_member_list_ cost time(ret=0, time_us=42) [2025-01-08 00:51:05.337716] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:51:05.337730] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:05.337743] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:05.337756] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=12] execute_gc cost time(ret=0, time_us=14) [2025-01-08 00:51:05.337763] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] ObSafeDestroyHandler start process [2025-01-08 00:51:05.338347] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918541C-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:05.338467] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.344044] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=47, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:05.344774] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=56, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:05.345757] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58381][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A451B-0-0] [lt=16][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18012, log_task={header:{begin_lsn:{lsn:2192311}, end_lsn:{lsn:2192433}, log_id:18012, min_scn:{val:1736326265146284000, v:0}, max_scn:{val:1736326265146284000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2192198}, prev_proposal_id:1, committed_end_lsn:{lsn:2192311}, data_checksum:1715074492, accum_checksum:2081098416, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326265243774, freeze_ts:1736326265243774, submit_ts:1736326265243784, flushed_ts:1736326265257717, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:13933}, fs_cb_begin_ts=1736326265345732, log_life_time=101958) [2025-01-08 00:51:05.345875] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17924, miss_cnt:0, cache_read_size:2192433, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:05.346662] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=75, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:05.346687] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=46, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:05.349157] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=49] current monitor number(seq_=-1) [2025-01-08 00:51:05.361938] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] no need to update min/max version(new_version={val:1736326264955075000, v:0}, min_version={val:1736326264955075000, v:0}, max_version={val:1736326264955075000, v:0}) [2025-01-08 00:51:05.377302] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=78] get number(ret=0, service_type_=0, range=1, base_id=1736326265376575000, start_id=1736326265376575000, end_id=1736326265376575001) [2025-01-08 00:51:05.385352] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=32] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:05.386835] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=36] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:05.386929] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326255577817000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:05.386956] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326255577817000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326255577817000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:05.392631] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=17] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:05.392720] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=28] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:05.392739] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=18] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=86) [2025-01-08 00:51:05.395725] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:05.395760] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-01-08 00:51:05.395771] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:05.395785] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:05.395798] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193769309415, flush_io_finished_ret_:0, flush_io_finished_round_:1924, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:05.405206] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:05.405251] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=33] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:05.405266] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=13] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:05.405273] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:05.405280] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:05.405288] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=93) [2025-01-08 00:51:05.405985] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:05.406048] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=48] temporary file module use no blocks [2025-01-08 00:51:05.406103] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=54] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:05.406124] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:05.406138] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193769319724, flush_io_finished_ret_:0, flush_io_finished_round_:3752, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:05.426299] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:05.431032] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:05.432671] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:05.434722] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58387][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC0-0-0] [lt=204] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326265529497, keepalive_ts:-1}) [2025-01-08 00:51:05.434750] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58387][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC0-0-0] [lt=27] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326265529497, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:05.434985] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=37] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=597683, clean_num=31457) [2025-01-08 00:51:05.436994] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] ====== tenant freeze timer task ====== [2025-01-08 00:51:05.437037] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=35] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70075580, tenant_id=1001) [2025-01-08 00:51:05.437067] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=20] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:05.437364] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=8, avg_log_batch_cnt=1, total_group_log_size=976, avg_group_log_size=122) [2025-01-08 00:51:05.437409] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=976) [2025-01-08 00:51:05.438652] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.439051] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58381][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37C1-0-0] [lt=57][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17911, log_task={header:{begin_lsn:{lsn:2180034}, end_lsn:{lsn:2180156}, log_id:17911, min_scn:{val:1736326265146284002, v:0}, max_scn:{val:1736326265146284002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2179912}, prev_proposal_id:1, committed_end_lsn:{lsn:2180034}, data_checksum:940009244, accum_checksum:2109833473, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326265338860, freeze_ts:1736326265338860, submit_ts:1736326265338867, flushed_ts:1736326265342650, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:3783}, fs_cb_begin_ts=1736326265439025, log_life_time=100165) [2025-01-08 00:51:05.441233] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB APPEND COST TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=88, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:05.441261] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=27] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=22712, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:05.441271] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:05.441278] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=-12258, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:05.441285] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=10542, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:05.443926] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=93] ====== tenant freeze timer task ====== [2025-01-08 00:51:05.444862] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326261282370, last_first_register_time_us=1736324454612308, register_time_us=1736326261282370, curr_time_us=1736326265444859) [2025-01-08 00:51:05.444978] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=73] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326265444937, last_submit_register_req_time_us=1736326265444937, register_parent_reason=1) [2025-01-08 00:51:05.444997] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:05.458835] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.459088] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.462346] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BEC6-0-0] [lt=13] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:05.468777] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58394][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67DA-0-0] [lt=32] init point thread id with(&point=0x5572d50e2fc0, idx_=1050, point=[thread id=58394, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58394) [2025-01-08 00:51:05.519001] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.538872] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=219][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.542169] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.544461] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.581483] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=43143782, this={dio_aligned_buf:{buf_write_offset:1203, buf_padding_size:2893, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:45412, truncate_used_ts:13155}, log_block_size:67108864, io_fd:{first_id:-1, second_id:160, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=43143168, buf_len=589, total_write_size=85169, total_write_size_after_dio=696320, ob_pwrite_used_ts=858370, count=149, each_pwrite_cost=5760) [2025-01-08 00:51:05.626518] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD4-0-0] [lt=26] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:05.626554] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD4-0-0] [lt=28] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=38) [2025-01-08 00:51:05.630246] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=606, get_gts_with_stc_cnt=21, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=21, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:05.630287] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=491, get_gts_with_stc_cnt=61, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=59, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:05.631345] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=79] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:05.634465] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:05.634568] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:05.635574] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:05.639782] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.640861] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B97-0-0] [lt=23] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:05.641057] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B99-0-0] [lt=27] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:05.642510] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5216, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:05.642542] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=687, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:05.646516] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1736326265646513, bytes: 0, bw: 0.000000 MB/s, add_ts: 1002372, add_bytes: 0 [2025-01-08 00:51:05.649667] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=48] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:05.650032] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:05.651934] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=49] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:05.652268] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:05.655442] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58381][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=37] worker thread exit(tenant_->id()=1002, workers_.get_size()=6) [2025-01-08 00:51:05.656234] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90FD-0-0] [lt=21] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:05.656256] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A90FD-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=72506, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:05.661047] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:05.666035] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3075) [2025-01-08 00:51:05.669545] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853A5-0-0] [lt=10] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:05.669590] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853A5-0-0] [lt=26] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:05.675487] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=30] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:05.689374] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB86-0-0] [lt=37][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193770603206, cache_obj->added_lc()=false, cache_obj->get_object_id()=13591, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:05.730935] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC395-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:05.738767] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37C7-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:05.738840] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37C7-0-0] [lt=46] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:05.738860] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37C7-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:05.739943] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.740135] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=46, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:05.740117] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=113, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:05.740501] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=83, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:05.740521] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:05.740535] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:05.740548] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:05.741606] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:05.741676] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=66] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:05.746582] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84C6-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:05.746635] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84C6-0-0] [lt=52] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=10653, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:05.746719] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:05.746757] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4523-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=23, avg_log_submit_to_flush_time=7185, avg_log_submit_to_slide_time=60421) [2025-01-08 00:51:05.748015] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=116] [PALF STAT WRITE LOG COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=4088, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:05.748041] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10343) [2025-01-08 00:51:05.748059] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=954, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:05.748086] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=34, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:05.755406] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38D7-0-0] [lt=52] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:05.755429] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38D7-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=22109, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:05.755439] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38D7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=21, avg_log_gen_to_freeze_time=13836, avg_log_gen_to_submit_time=13918, avg_log_submit_to_flush_time=5093, avg_log_submit_to_slide_time=8191) [2025-01-08 00:51:05.755592] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=11054, hit_cnt=21, read_cnt=21, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:05.768440] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=129] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:05.772566] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=37] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:05.777636] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50915][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1609BCBE-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1736326265774978000, start_id=1736326265774978000, end_id=1736326265774978001) [2025-01-08 00:51:05.777828] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ACAB-0-0] [lt=58] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.719243288040161133e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=385, Throttle Trigger(MB)=231, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.905212950641981706e+02) [2025-01-08 00:51:05.784365] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=39] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326265128357003, v:0}, new_max_version={val:1736326265128357003, v:0}, last_min_version={val:1736326264428511005, v:0}, last_max_version={val:1736326264428511005, v:0}, total_time_us=8304, delta_us=656005) [2025-01-08 00:51:05.784399] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326265128357003, v:0}, min_version={val:1736326265128357003, v:0}, max_version={val:1736326265128357003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:05.785103] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=43] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:05.792183] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:05.802959] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D02F-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:05.806366] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1609BCBE-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1736326265805677000, start_id=1736326265805677000, end_id=1736326265805677001) [2025-01-08 00:51:05.826837] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:05.831629] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:05.836048] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:05.838341] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A73-0-0] [lt=16] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326265838324}) [2025-01-08 00:51:05.840066] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=85][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.841333] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285966-0-0] [lt=29] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:05.844089] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5201, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:05.844121] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=378, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:05.844856] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5311, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:05.844882] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=250, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:05.844994] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=144, avg_log_submit_to_flush_time=5621, avg_log_submit_to_slide_time=8162) [2025-01-08 00:51:05.845097] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=34, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:05.845569] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5209345347, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:05.845674] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=43] recycle one thread(this=0x2ad8885abcf0, total=5, remain=4) [2025-01-08 00:51:05.845689] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5209345347, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:05.847007] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=4987, avg_group_log_size=383) [2025-01-08 00:51:05.847033] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4987) [2025-01-08 00:51:05.847047] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:05.847254] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84CA-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=27, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:05.847344] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84CA-0-0] [lt=87] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=9325, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:05.848866] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:05.851340] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4525-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:05.851909] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285966-0-0] [lt=26] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265851881, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265851881, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265851881, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:05.851957] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285966-0-0] [lt=46] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326265838324}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265851881, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265851881, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326265851881, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:05.859382] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84CB-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=46, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:05.859411] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84CB-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=47085, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:05.859472] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:05.860334] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1003}, {id:1002}]) [2025-01-08 00:51:05.861774] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=44] ls blacklist refresh finish(cost_time=25137) [2025-01-08 00:51:05.895498] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=93] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:05.897257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=92] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.897301] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.897327] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.897350] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:05.915263] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=39] PNIO [ratelimit] time: 1736326265915258, bytes: 870506447, bw: 0.347459 MB/s, add_ts: 1007335, add_bytes: 367010 [2025-01-08 00:51:05.939391] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37CB-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=4567, avg_log_submit_to_slide_time=66875) [2025-01-08 00:51:05.939432] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37CB-0-0] [lt=39][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17916, log_task={header:{begin_lsn:{lsn:2180644}, end_lsn:{lsn:2180766}, log_id:17916, min_scn:{val:1736326265819346000, v:0}, max_scn:{val:1736326265819346000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2180522}, prev_proposal_id:1, committed_end_lsn:{lsn:2180644}, data_checksum:940009244, accum_checksum:1539127517, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326265839074, freeze_ts:1736326265839074, submit_ts:1736326265839088, flushed_ts:1736326265848969, gen_to_freeze cost time:0, gen_to_submit cost time:14, submit_to_flush cost time:9881}, fs_cb_begin_ts=1736326265939366, log_life_time=100292) [2025-01-08 00:51:05.939516] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=152] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:05.939556] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17827, miss_cnt:0, cache_read_size:2180766, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:05.940307] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:05.948890] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4527-0-0] [lt=34] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:05.963109] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326265715468000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=247630, in_cluster_service=true, cluster_version={val:1736326265663496000, v:0}, min_cluster_version={val:1736326265663496000, v:0}, max_cluster_version={val:1736326265663496000, v:0}, get_cluster_version_err=0, cluster_version_delta=299602, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36402, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=520460085, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:05.974843] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=6] Garbage Collector is running(seq_=193, gc_interval=10000000) [2025-01-08 00:51:05.974963] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=48] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:05.974994] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=16] gc_check_member_list_ cost time(ret=0, time_us=101) [2025-01-08 00:51:05.975009] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:05.976957] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:05.976987] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=22] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:05.977004] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=15] execute_gc cost time(ret=0, time_us=19) [2025-01-08 00:51:05.977013] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyHandler start process [2025-01-08 00:51:05.978165] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=10181, avg_group_log_size=535) [2025-01-08 00:51:05.978255] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=85] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10343) [2025-01-08 00:51:05.980424] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F2-0-0] [lt=49] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:05.980508] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F2-0-0] [lt=65] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326265819346001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326265819346001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:06.001295] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=19] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:06.031838] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:06.036683] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:06.040347] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.045729] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1912, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:06.048350] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:06.054266] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58387][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=14] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:51:06.058876] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D37-0-0] [lt=45] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:06.062579] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C91-0-0] [lt=47][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:06.066001] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C91-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:06.125569] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50503][T1002_L0_G0][T1002][YB42C0A89F8B-00062B2D1C3BBAE6-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326266125549000, start_id=1736326266125549000, end_id=1736326266125549001) [2025-01-08 00:51:06.131028] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326266131006], gts:1736326266130647000, latest_srr:[mts=1736326266131006]}) [2025-01-08 00:51:06.131078] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=47] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:06.131106] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326266131092], gts:1736326266130647000, latest_srr:[mts=1736326266131092]}) [2025-01-08 00:51:06.131424] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326266131391], gts:1736326266130647000, latest_srr:[mts=1736326266131391]}) [2025-01-08 00:51:06.134857] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:06.140560] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.148183] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1003, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:17919, max_lsn:2181132, max_scn:{val:1736326266133958000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2181132}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2181010}, last_submit_end_lsn:{lsn:2181132}, last_submit_log_id:17919, last_submit_log_pid:1, max_flushed_lsn:{lsn:2181010}, max_flushed_end_lsn:{lsn:2181132}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2181132}, last_slide_log_id:17919, last_slide_scn:{val:1736326266133958000, v:0}, last_slide_lsn:{lsn:2181010}, last_slide_end_lsn:{lsn:2181132}, last_slide_log_pid:1, last_slide_log_accum_checksum:200919321, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924fd6530}, [StateMgr]={this:0x2ad924fda830, self:"192.168.159.137:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.139:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:17920, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad924fe12f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326265444937, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326265444937, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924fd7c70}, [ModeMgr]={palf_id:1003, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436825152001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/meta", dir_fd:181, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924fdca38, flashback_version:0}, log_storage:{log_tail:{lsn:2181132}, readable_log_tail:{lsn:2181132}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64923636, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, palf_epoch:4, last_purge_throttling_ts:-1, this:0x2ad924fdaf70}, [Reconfirm]={palf_id:1003, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924fda9b0}) [2025-01-08 00:51:06.148568] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=380][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:06.148587] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:06.148599] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:06.148609] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] current log_task status(palf_id=1003, self="192.168.159.137:2882", log_id=17920, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:51:06.150007] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:06.150349] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:06.152979] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:06.161357] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:06.161589] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [46346][T1_ArcFetcher][T1][YB42C0A89F89-00062B2D1678458E-0-0] [lt=23] ObArchiveFetcher is running(thread_index=0) [2025-01-08 00:51:06.165900] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:06.166461] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58394][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=40] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:51:06.175710] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=24] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:06.181116] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10342) [2025-01-08 00:51:06.187174] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58400][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:06.187407] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:51:06.202344] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8614, offset=2444212}, end_cursor:ObLogCursor{file_id=1, log_id=8615, offset=2444517}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89478c050, len:1100, log_data_len:305, seq:8614, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:06.203059] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E49-0-0] [lt=104] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:06.203129] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E49-0-0] [lt=47] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326266125549000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326266125549000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:06.208983] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E49-0-0] [lt=53] get number(ret=0, service_type_=0, range=1, base_id=1736326266203795000, start_id=1736326266203795000, end_id=1736326266203795001) [2025-01-08 00:51:06.213462] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29C-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:06.213523] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29C-0-0] [lt=45] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326265815330002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326265815330002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:06.214331] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=57] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8615, offset=2444517}, end_cursor:ObLogCursor{file_id=1, log_id=8616, offset=2444814}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894788050, len:795, log_data_len:297, seq:8615, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:06.215062] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E49-0-0] [lt=126] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:06.215100] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E49-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326266130647000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326266130647000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:06.225909] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=55] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8616, offset=2444814}, end_cursor:ObLogCursor{file_id=1, log_id=8617, offset=2445111}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894790050, len:498, log_data_len:297, seq:8616, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:06.227699] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=26] gc stale ls task succ [2025-01-08 00:51:06.232042] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:06.237310] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:06.240843] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.241270] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:06.244156] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:06.244341] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=183][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:06.248528] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84D5-0-0] [lt=42] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2457) [2025-01-08 00:51:06.248868] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D16EBBDA4-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326266245500000, start_id=1736326266245500000, end_id=1736326266245500001) [2025-01-08 00:51:06.249375] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:06.249425] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48817, [delay/us]:prepare:7, schedule:251, submit:124, rt:1594, total:1977) [2025-01-08 00:51:06.249440] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:06.249495] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48817, [delay/us]: prepare:7, schedule:251, submit:124, device:1594, total:1977) [2025-01-08 00:51:06.249508] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:06.249547] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4742, iops:19, bw:89499, [delay/us]:prepare:9, schedule:165, submit:263, rt:1623, total:2061) [2025-01-08 00:51:06.249555] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:06.249694] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=123] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4743, avg_iops:19, avg_bw:89490, [delay/us]: prepare:9, schedule:165, submit:263, device:1623, total:2061) [2025-01-08 00:51:06.249708] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:06.249732] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:12288 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12203, [delay/us]:prepare:7, schedule:55, submit:1, rt:1423, total:1487) [2025-01-08 00:51:06.249777] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:3, hold_mem:24576 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:38, bw:154580, [delay/us]:prepare:5, schedule:246, submit:266, rt:2101, total:2619) [2025-01-08 00:51:06.249788] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1002, ref_cnt=10, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49996, free_result_cnt=49996, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:06.249819] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:3, avg_bw:12203, [delay/us]: prepare:7, schedule:55, submit:1, device:1423, total:1487) [2025-01-08 00:51:06.249846] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:38, avg_bw:154575, [delay/us]: prepare:5, schedule:246, submit:266, device:2101, total:2619) [2025-01-08 00:51:06.249861] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:06.249909] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4067, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1307, total:1317) [2025-01-08 00:51:06.249936] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:06.249967] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4067, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1307, total:1317) [2025-01-08 00:51:06.250086] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=105] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:06.250103] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:06.250112] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:06.250120] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:06.250128] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=351kB/s, net_out=321kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:06.250599] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A452D-0-0] [lt=19] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:06.250748] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A452D-0-0] [lt=129][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18021, log_task={header:{begin_lsn:{lsn:2193409}, end_lsn:{lsn:2193531}, log_id:18021, min_scn:{val:1736326266141084000, v:0}, max_scn:{val:1736326266141084000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2193287}, prev_proposal_id:1, committed_end_lsn:{lsn:2193409}, data_checksum:1715074492, accum_checksum:1153450415, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326266147445, freeze_ts:1736326266147445, submit_ts:1736326266147451, flushed_ts:1736326266150115, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:2664}, fs_cb_begin_ts=1736326266250735, log_life_time=103290) [2025-01-08 00:51:06.251162] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Slow write(duration=25035, write_len=8192) [2025-01-08 00:51:06.251178] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=13] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8617, offset=2445111}, end_cursor:ObLogCursor{file_id=1, log_id=8618, offset=2445408}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894794050, len:4297, log_data_len:297, seq:8617, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:06.251855] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2160383}], scn_array=[cnt:1, {val:1736326266130647001, v:0}], curr_size=122, accum_size=1342, time_cost=10412) [2025-01-08 00:51:06.251929] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:06.252972] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:149) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C29C-0-0] [lt=15] transaction rpc statistics(total_rt=9825, total_process=33, avg_rt=288) [2025-01-08 00:51:06.253632] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2457) [2025-01-08 00:51:06.253704] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=2527, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:06.253725] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5410) [2025-01-08 00:51:06.253741] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=1123, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:06.253766] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:06.256014] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=92] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3104, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:06.256041] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:06.256057] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=338, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:06.257865] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50625][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C9BB5C7-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1736326266254219000, start_id=1736326266254219000, end_id=1736326266254219001) [2025-01-08 00:51:06.258046] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=42, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:06.259615] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B4B-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:06.259635] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B4B-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=7667, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:06.259645] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B4B-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=932, avg_log_gen_to_submit_time=1004, avg_log_submit_to_flush_time=3693, avg_log_submit_to_slide_time=6662) [2025-01-08 00:51:06.260301] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5410, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:06.260427] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87067-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:06.260484] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:06.260496] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:06.286109] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:51:06.286955] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=110] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:06.287056] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=94] temporary file module use no blocks [2025-01-08 00:51:06.287121] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:06.287283] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=158] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:06.287303] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193770200323, flush_io_finished_ret_:0, flush_io_finished_round_:1929, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:06.299431] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:06.317418] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA8-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:06.317477] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA8-0-0] [lt=54] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:06.318265] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=214] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:06.319201] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=69] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.319432] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.331872] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:06.340424] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918541D-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:06.341041] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=248][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.341738] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:332K/78:269435 disk:332K/78:269435 t1m2:56K/13:223547 t1m2g0:56K/13:223547 t1002m2:180K/44:312415 t1002m2g0:180K/44:312415 t1001m2:96K/21:207788 t1001m2g0:96K/21:207788 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:06.345740] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:06.347599] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=55] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=130, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:06.347626] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:06.349368] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=86] current monitor number(seq_=-1) [2025-01-08 00:51:06.350374] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17934, miss_cnt:0, cache_read_size:2193653, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:06.379258] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=88, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:06.381370] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=111] get number(ret=0, service_type_=0, range=1, base_id=1736326266381321000, start_id=1736326266381321000, end_id=1736326266381321001) [2025-01-08 00:51:06.395976] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:06.396018] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=34] temporary file module use no blocks [2025-01-08 00:51:06.396029] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:06.396043] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:06.396057] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193770309670, flush_io_finished_ret_:0, flush_io_finished_round_:1925, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:06.406340] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:06.406376] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] temporary file module use no blocks [2025-01-08 00:51:06.406449] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=71] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:06.406461] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:06.406473] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193770320008, flush_io_finished_ret_:0, flush_io_finished_round_:3753, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:06.426443] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=37] gc stale ls task succ [2025-01-08 00:51:06.432266] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=90] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:06.437729] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:06.441158] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=137] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:06.441395] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=234][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.441537] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=2335, avg_group_log_size=233) [2025-01-08 00:51:06.441563] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2335) [2025-01-08 00:51:06.449838] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=55, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:06.458772] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.459048] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.519368] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=428] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.541271] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.542646] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=185] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.544395] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=77] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.579423] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ACB7-0-0] [lt=46] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326266579273, can_tenant_elr:true}) [2025-01-08 00:51:06.587777] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58400][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2C51-0-0] [lt=107] init point thread id with(&point=0x5572d50e3140, idx_=1056, point=[thread id=58400, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58400) [2025-01-08 00:51:06.588752] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB18F-0-0] [lt=62] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:06.588788] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB18F-0-0] [lt=30] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:06.588818] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB18F-0-0] [lt=28] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:06.633301] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=30] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1179600, replace_num=15728) [2025-01-08 00:51:06.633355] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:06.636743] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:06.637793] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE829-0-0] [lt=73] get number(ret=0, service_type_=0, range=1, base_id=1736326266637045000, start_id=1736326266637045000, end_id=1736326266637045001) [2025-01-08 00:51:06.638390] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:06.641901] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.641973] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE828-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1736326266640901000, start_id=1736326266640901000, end_id=1736326266640901001) [2025-01-08 00:51:06.643874] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F4C0-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326266640901000, start_id=1736326266640901000, end_id=1736326266640901001) [2025-01-08 00:51:06.647739] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2809, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:06.647773] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=530, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:06.652634] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=53] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:06.653973] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1736326266653970, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007457, add_bytes: 0 [2025-01-08 00:51:06.654020] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:06.654774] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58393][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD88-0-0] [lt=38] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326266748159, keepalive_ts:-1}) [2025-01-08 00:51:06.654803] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58393][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD88-0-0] [lt=28] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326266748159, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:06.656415] INFO [LIB] stat (utility.h:1210) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A911A-0-0] [lt=34] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=37, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:06.656441] INFO [LIB] stat (utility.h:1210) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A911A-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16466, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:06.659949] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:06.661906] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:06.664655] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:06.667673] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58244][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=24] worker thread exit(tenant_->id()=1002, workers_.get_size()=5) [2025-01-08 00:51:06.667821] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67E7-0-0] [lt=117] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=5289) [2025-01-08 00:51:06.676082] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:06.677409] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=215] get_min_unreplayed_log_info(lsn={lsn:2194019}, scn={val:1736326266141084005, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326266552717, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2194019}, next_to_submit_scn_:{val:1736326266141084005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2193897}, end_lsn:{lsn:2194019}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2194141}, readable_log_tail:{lsn:2194141}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64910627, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326266141084004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326266141084004, v:0}, data_checksum:3348837161, flag:58886}}, init_mode_version:0, accumulate_checksum:3967272563, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326266141084004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17937, miss_cnt:0, cache_read_size:2194019, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:06.706225] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [50457][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=93] XA scheduler heartbeat task statistics(avg_time=2) [2025-01-08 00:51:06.718297] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB88-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193771632127, cache_obj->added_lc()=false, cache_obj->get_object_id()=13592, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:06.743509] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.743692] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=122, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:06.744317] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=73, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:06.744378] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:06.744403] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:06.744479] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:06.746037] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:06.746089] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:06.749564] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50507][T1002_L0_G0][T1002][YB42C0A89F8A-00062B2D2269B9E4-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326266749524000, start_id=1736326266749524000, end_id=1736326266749524001) [2025-01-08 00:51:06.751633] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37DB-0-0] [lt=28] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:06.751700] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37DB-0-0] [lt=56] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:06.751719] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37DB-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:06.753283] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4537-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=9, avg_log_submit_to_flush_time=2893, avg_log_submit_to_slide_time=61141) [2025-01-08 00:51:06.753394] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=34] ====== tenant freeze timer task ====== [2025-01-08 00:51:06.754097] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9120-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:06.754112] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9120-0-0] [lt=15] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6247, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:06.754777] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708514D-0-0] [lt=44] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:06.757150] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=38, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:06.759416] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:06.760953] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=2427, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:06.760978] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10307) [2025-01-08 00:51:06.760997] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=338, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:06.761023] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=41] [PALF STAT FLUSH CB COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=46, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:06.762578] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38EA-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:06.762622] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38EA-0-0] [lt=42] [PALF STAT LOG LIFE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=5228, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:06.762637] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38EA-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=541, avg_log_gen_to_submit_time=646, avg_log_submit_to_flush_time=2813, avg_log_submit_to_slide_time=4582) [2025-01-08 00:51:06.762749] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10307, hit_cnt=19, read_cnt=19, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:06.772733] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:06.779409] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ACBA-0-0] [lt=127] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.695395278930664062e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=384, Throttle Trigger(MB)=230, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=8.003027966626933676e+02) [2025-01-08 00:51:06.785338] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:06.791755] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC398-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:06.794152] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326266579273001, v:0}, new_max_version={val:1736326266579273001, v:0}, last_min_version={val:1736326266378990000, v:0}, last_max_version={val:1736326266378990000, v:0}, total_time_us=13908, delta_us=214876) [2025-01-08 00:51:06.794190] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326266579273001, v:0}, min_version={val:1736326266579273001, v:0}, max_version={val:1736326266579273001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:06.794755] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=63] get_max_applied_scn(scn={val:1736326266631374000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18665502}, palf_committed_end_scn_:{val:1736326266749524000, v:0}, last_check_scn_:{val:1736326266749524000, v:0}, max_applied_cb_scn_:{val:1736326266631374000, v:0}}) [2025-01-08 00:51:06.794792] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18665502}, readable_log_tail:{lsn:18665502}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48439266, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:06.794937] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=122] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:06.816176] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D032-0-0] [lt=42][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:06.827125] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=37] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:06.828973] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=49] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:06.829012] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=27] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=177) [2025-01-08 00:51:06.829075] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=18] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:06.829103] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=19] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:06.829116] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=192) [2025-01-08 00:51:06.833549] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:06.838866] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=119] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:06.843743] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.846330] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2728, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:06.846389] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=131, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:06.846737] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5213085076, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:06.846822] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:06.846839] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8bf5f1b90) [2025-01-08 00:51:06.846848] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5213085076, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:06.853613] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4539-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:06.857772] INFO [LIB] stat (utility.h:1210) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9121-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:06.857909] INFO [LIB] stat (utility.h:1210) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9121-0-0] [lt=135] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7489, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:06.860928] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1001}, {id:1002}]) [2025-01-08 00:51:06.863303] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [46323][T1_RFLWorker][T1][YB42C0A89F89-00062B2D1618458E-0-0] [lt=112] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:51:06.864436] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D73E-0-0] [lt=51] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:06.864483] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D73E-0-0] [lt=47] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=61732, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:06.864523] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:06.867087] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=73] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4972, avg_group_log_size=452) [2025-01-08 00:51:06.867131] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4508) [2025-01-08 00:51:06.867209] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=74] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:06.895702] INFO [STORAGE.COMPACTION] schedule_all_tablets_medium (ob_tenant_tablet_scheduler.cpp:1597) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84ED6-0-0] [lt=59] start schedule all tablet merge(merge_version=1) [2025-01-08 00:51:06.896301] INFO [STORAGE] get_adaptive_merge_reason (ob_partition_merge_policy.cpp:1719) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84ED6-0-0] [lt=83] Check tablet adaptive merge reason(ret=0, ls_id={id:1}, tablet_id={id:60476}, reason=0, tablet_analyzer={tablet_stat:{ls_id:1, tablet_id:60476, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, total_tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_small_tenant:false, boost_factor:"1.000000000000000000e+00"}, crazy_medium_flag=false) [2025-01-08 00:51:06.917052] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.917116] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.917173] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.917231] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:06.918230] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=134] ====== check clog disk timer task ====== [2025-01-08 00:51:06.918391] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=150] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:06.919461] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=81] PNIO [ratelimit] time: 1736326266919457, bytes: 870722960, bw: 0.205619 MB/s, add_ts: 1004199, add_bytes: 216513 [2025-01-08 00:51:06.969416] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:06.969619] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=32] LogLoopThread round_cost_time(us)(round_cost_time=78) [2025-01-08 00:51:06.970262] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:06.970402] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=96] ====== check clog disk timer task ====== [2025-01-08 00:51:06.970423] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=17] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:06.971433] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] clock out of order(cur_ts=1736326266971431, cur_ts_=1736326266928358, delta=43073) [2025-01-08 00:51:06.972213] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2181864}], scn_array=[cnt:1, {val:1736326266637045001, v:0}], curr_size=122, accum_size=1220, time_cost=43963) [2025-01-08 00:51:06.972275] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=61] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:06.972963] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3512, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:06.973014] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=624, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:06.973076] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:06.977830] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326266130647003, v:0}, base_version={val:1736326266130647003, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326266631374001, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326266977811, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326266130647003, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326266944875, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326266133958005, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326266925928, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:06.977876] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44] no need to update min/max version(new_version={val:1736326266130647003, v:0}, min_version={val:1736326266130647003, v:0}, max_version={val:1736326266130647003, v:0}) [2025-01-08 00:51:06.980494] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=84] no need to update min/max version(new_version={val:1736326266579273001, v:0}, min_version={val:1736326266579273001, v:0}, max_version={val:1736326266579273001, v:0}) [2025-01-08 00:51:07.026016] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C89C282-0-0] [lt=228] get number(ret=0, service_type_=0, range=1, base_id=1736326266980863000, start_id=1736326266980863000, end_id=1736326266980863001) [2025-01-08 00:51:07.026196] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C89C282-0-0] [lt=49] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=10265, avg_group_log_size=540) [2025-01-08 00:51:07.026219] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C89C282-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10307) [2025-01-08 00:51:07.026410] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A453B-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:07.033135] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=194, gc_interval=10000000) [2025-01-08 00:51:07.033254] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=27] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:07.033274] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=17] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:07.033802] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:07.035404] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=88] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:07.035404] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58379][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D73F-0-0] [lt=24] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=20, avg_log_submit_to_flush_time=7281, avg_log_submit_to_slide_time=19895) [2025-01-08 00:51:07.035444] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17837, miss_cnt:0, cache_read_size:2181986, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:07.035435] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58379][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D73F-0-0] [lt=31][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17926, log_task={header:{begin_lsn:{lsn:2181864}, end_lsn:{lsn:2181986}, log_id:17926, min_scn:{val:1736326266637045001, v:0}, max_scn:{val:1736326266637045001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2181742}, prev_proposal_id:1, committed_end_lsn:{lsn:2181864}, data_checksum:940009244, accum_checksum:1591647518, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326266928004, freeze_ts:1736326266928005, submit_ts:1736326266928020, flushed_ts:1736326266972460, gen_to_freeze cost time:1, gen_to_submit cost time:16, submit_to_flush cost time:44440}, fs_cb_begin_ts=1736326267035322, log_life_time=107318) [2025-01-08 00:51:07.035740] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9123-0-0] [lt=31] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=69, avg_log_gen_to_submit_time=199, avg_log_submit_to_flush_time=4182, avg_log_submit_to_slide_time=11764) [2025-01-08 00:51:07.037503] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:51:07.039467] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=89] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:07.067552] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4015] clock out of order(cur_ts=1736326267067548, cur_ts_=1736326267045947, delta=21601) [2025-01-08 00:51:07.069814] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=51] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:07.069894] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=78][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:07.075934] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C92-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:07.077566] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=23] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:51:07.077660] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=86] gc_check_member_list_ cost time(ret=0, time_us=44498) [2025-01-08 00:51:07.077681] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=13] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:51:07.079860] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:07.080082] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=17] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:07.080113] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=93) [2025-01-08 00:51:07.081565] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=21] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:07.082755] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C92-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:07.082872] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:07.084407] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:07.084565] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=97] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:07.084605] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=38] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:07.084635] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=29] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:07.084653] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=17] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:07.084668] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=14] execute_gc cost time(ret=0, time_us=104) [2025-01-08 00:51:07.084678] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyHandler start process [2025-01-08 00:51:07.084687] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-01-08 00:51:07.084702] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-01-08 00:51:07.087369] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=63][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2161359}], scn_array=[cnt:1, {val:1736326266749524002, v:0}], curr_size=122, accum_size=976, time_cost=10017) [2025-01-08 00:51:07.087995] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84244772}], scn_array=[cnt:1, {val:1736326266866279001, v:0}], curr_size=122, accum_size=3255, time_cost=10676) [2025-01-08 00:51:07.100672] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=8] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84245480}, readable_log_tail:{lsn:84245480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49964056, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:07.100897] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=224] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:07.100931] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=15] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:07.104067] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2066, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:07.106401] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=51] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:07.132840] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=64] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:07.141376] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185794-0-0] [lt=33][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193772055205, cache_obj->added_lc()=false, cache_obj->get_object_id()=13593, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:07.142723] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:07.144151] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185795-0-0] [lt=125][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:07.146325] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185797-0-0] [lt=87][errcode=-4201] hashtable not init [2025-01-08 00:51:07.153946] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:07.154149] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=119] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:07.160289] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:07.163754] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:07.169990] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:07.174724] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:07.176286] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58393][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=13] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:51:07.176284] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=34] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:07.189592] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=41] ====== check clog disk timer task ====== [2025-01-08 00:51:07.189629] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=27] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:07.191439] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58385][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2C57-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9982) [2025-01-08 00:51:07.195738] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:07.199290] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58339][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=34] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:51:07.205206] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A453F-0-0] [lt=19][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18030, log_task={header:{begin_lsn:{lsn:2194507}, end_lsn:{lsn:2194629}, log_id:18030, min_scn:{val:1736326266640901003, v:0}, max_scn:{val:1736326266640901003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2194385}, prev_proposal_id:1, committed_end_lsn:{lsn:2194507}, data_checksum:1715074492, accum_checksum:3740797114, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326267091108, freeze_ts:1736326267091108, submit_ts:1736326267091117, flushed_ts:1736326267094080, gen_to_freeze cost time:0, gen_to_submit cost time:9, submit_to_flush cost time:2963}, fs_cb_begin_ts=1736326267205192, log_life_time=114084) [2025-01-08 00:51:07.207326] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2198725, this={dio_aligned_buf:{buf_write_offset:3391, buf_padding_size:705, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:26348, truncate_used_ts:5407}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2195456, buf_len=122, total_write_size=11928, total_write_size_after_dio=413696, ob_pwrite_used_ts=540181, count=98, each_pwrite_cost=5512) [2025-01-08 00:51:07.211623] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] ====== traversal_flush timer task ====== [2025-01-08 00:51:07.211726] INFO [MDS] flush (mds_table_mgr.cpp:195) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853A8-0-0] [lt=24] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326267132611000, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326263438968003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326280087570000, latest_log_ts:{val:1736326263438968001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.211854] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853A8-0-0] [lt=94] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326261935253001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326263438968003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.212226] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853A8-0-0] [lt=216] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326262002109001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326263438968003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.212321] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853A8-0-0] [lt=64] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326261717746002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326263438968003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.212440] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853A8-0-0] [lt=115] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:07.228070] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=88] gc stale ls task succ [2025-01-08 00:51:07.233990] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:07.239980] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:07.244080] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:07.244108] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:07.246359] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8706A-0-0] [lt=9][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:07.246405] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:07.246417] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:07.266283] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x5572d527f900, last_check_time:1736326267266278, last_idle_time:15053760825, last_thread_count:4}, *pool={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15053760825, tenant_id:500}, qsize=-4, current_idle_time=15053760825) [2025-01-08 00:51:07.266331] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=47] [thread pool stat](pool_stat={pool:0x2ad8885abcf0, last_check_time:1736326267266330, last_idle_time:5214271970, last_thread_count:4}, *pool={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5214271970, tenant_id:500}, qsize=-3, current_idle_time=5214271970) [2025-01-08 00:51:07.266344] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] [thread pool stat](pool_stat={pool:0x2ad8885acaf0, last_check_time:1736326267266344, last_idle_time:3763385225, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885acaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3763385225, tenant_id:500}, qsize=-1, current_idle_time=3763385225) [2025-01-08 00:51:07.266504] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=158] [thread pool stat](pool_stat={pool:0x2ad8881fc3b0, last_check_time:1736326267266502, last_idle_time:14984455707, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:14984455707, tenant_id:1}, qsize=-4, current_idle_time=14984455707) [2025-01-08 00:51:07.266515] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x2ad8885ad1f0, last_check_time:1736326267266515, last_idle_time:3675479452, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885ad1f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3675479452, tenant_id:500}, qsize=-1, current_idle_time=3675479452) [2025-01-08 00:51:07.266525] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8c0eb0e70, last_check_time:1736326267266525, last_idle_time:7774618887, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7774618887, tenant_id:1002}, qsize=-4, current_idle_time=7774618887) [2025-01-08 00:51:07.266534] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8f90b0e70, last_check_time:1736326267266534, last_idle_time:7744962980, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:7744962980, tenant_id:1001}, qsize=-3, current_idle_time=7744962980) [2025-01-08 00:51:07.267306] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84F2-0-0] [lt=139] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1643) [2025-01-08 00:51:07.269940] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=24] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:07.269965] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:07.270275] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E7-0-0] [lt=9] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:07.270302] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E7-0-0] [lt=23] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:07.270387] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:07.270567] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:07.274857] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=99] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8618, offset=2445408}, end_cursor:ObLogCursor{file_id=1, log_id=8619, offset=2445713}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89479c050, len:4000, log_data_len:305, seq:8618, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:07.275075] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=93] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:07.276095] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:07.278349] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:07.278456] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:43787, [delay/us]:prepare:17, schedule:928, submit:44, rt:2730, total:3721) [2025-01-08 00:51:07.278476] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:07.278530] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:43784, [delay/us]: prepare:17, schedule:928, submit:44, device:2730, total:3721) [2025-01-08 00:51:07.278544] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:07.278621] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:238, bw:975203, [delay/us]:prepare:1, schedule:106, submit:0, rt:1167, total:1275) [2025-01-08 00:51:07.278645] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4388, iops:14, bw:59706, [delay/us]:prepare:6, schedule:1217, submit:174, rt:1919, total:3317) [2025-01-08 00:51:07.278655] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1001, ref_cnt=5, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49998, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=59, ops=3, iops=62, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:07.278686] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:239, avg_bw:975235, [delay/us]: prepare:1, schedule:106, submit:0, device:1167, total:1275) [2025-01-08 00:51:07.278702] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4389, avg_iops:14, avg_bw:59707, [delay/us]: prepare:6, schedule:1217, submit:174, device:1919, total:3317) [2025-01-08 00:51:07.278712] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:07.278735] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:6144, iops:2, bw:11941, [delay/us]:prepare:3, schedule:32, submit:0, rt:13372, total:13408) [2025-01-08 00:51:07.278820] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=65] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4198, iops:39, bw:163202, [delay/us]:prepare:14, schedule:740, submit:134, rt:3482, total:4372) [2025-01-08 00:51:07.278833] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:07.278864] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:6144, avg_iops:2, avg_bw:11941, [delay/us]: prepare:3, schedule:32, submit:0, device:13372, total:13408) [2025-01-08 00:51:07.278880] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4198, avg_iops:39, avg_bw:163197, [delay/us]: prepare:14, schedule:740, submit:134, device:3482, total:4372) [2025-01-08 00:51:07.278891] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:07.278911] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3980, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:697, total:705) [2025-01-08 00:51:07.278909] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4A-0-0] [lt=21] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:07.278935] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:07.278962] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3980, [delay/us]: prepare:0, schedule:0, submit:600000000, device:697, total:705) [2025-01-08 00:51:07.278947] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4A-0-0] [lt=20] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326267132611000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326267132611000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:07.278980] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:07.278994] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:07.279003] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:07.279010] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:07.279029] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SERVER] net_in=191kB/s, net_out=184kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:07.283241] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58402][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:07.283357] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42] worker thread created(tenant_->id()=1002, group_id_=1, token=5) [2025-01-08 00:51:07.284113] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4830, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:07.284143] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84245480}], scn_array=[cnt:1, {val:1736326267132611000, v:0}], curr_size=122, accum_size=3963, time_cost=11443) [2025-01-08 00:51:07.284443] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=298] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3963) [2025-01-08 00:51:07.284479] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=779, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:07.284940] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F3-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:07.284977] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F3-0-0] [lt=27] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:07.285003] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F3-0-0] [lt=19] get_min_unreplayed_log_info(lsn={lsn:2182108}, scn={val:1736326267189859001, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326267265361, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2182108}, next_to_submit_scn_:{val:1736326267189859001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2181986}, end_lsn:{lsn:2182108}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2182108}, readable_log_tail:{lsn:2182108}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64922660, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326267189859000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326267189859000, v:0}, data_checksum:2124447694, flag:36723}}, init_mode_version:0, accumulate_checksum:3273801484, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326267189859000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17838, miss_cnt:0, cache_read_size:2182108, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:07.286546] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F3-0-0] [lt=1535] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326267189859000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326267189859000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:07.287931] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=38, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:07.289492] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B55-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:07.289516] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B55-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12475, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:07.289527] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B55-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=1540, avg_log_gen_to_submit_time=1610, avg_log_submit_to_flush_time=6029, avg_log_submit_to_slide_time=10864) [2025-01-08 00:51:07.290095] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=212] gc stale ls task succ [2025-01-08 00:51:07.291454] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3963, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:07.291919] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] ====== checkpoint timer task ====== [2025-01-08 00:51:07.292841] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:07.292957] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=114] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:07.293032] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=29] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326264665540004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326280087570000, latest_log_ts:{val:1736326263438968001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.293135] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=98] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:07.293159] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:07.293234] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=71] get rec log scn(service_type_=0, rec_log_ts={val:1736326263438968001, v:0}) [2025-01-08 00:51:07.293248] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:07.293286] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:07.293310] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:07.293324] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:07.293344] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:07.293365] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:07.294410] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:07.294465] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=53] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:07.294508] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=20] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326261935253001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326263438968003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.294588] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=77] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:07.294603] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:07.294615] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:07.294628] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:07.294663] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2161725}, scn={val:1736326267132611000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:07.295069] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:51:07.294965] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58403][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:07.295363] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58403][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67ED-0-0] [lt=218] init point thread id with(&point=0x5572d50e3200, idx_=1059, point=[thread id=58403, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58403) [2025-01-08 00:51:07.298631] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=46] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8619, offset=2445713}, end_cursor:ObLogCursor{file_id=1, log_id=8620, offset=2446010}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894798050, len:3695, log_data_len:297, seq:8619, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:07.298783] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43154955}], scn_array=[cnt:1, {val:1736326267281366000, v:0}], curr_size=1122, accum_size=4268, time_cost=13936) [2025-01-08 00:51:07.299832] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=37] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:07.302600] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4541-0-0] [lt=80] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:07.304890] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3537, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:07.304918] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:07.304971] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=962, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:07.306349] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:07.306811] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326267132611000, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:07.306854] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:07.306876] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:07.307791] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:07.307833] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=42] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:07.307885] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326262002109001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326263438968003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.307972] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=84] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:07.307990] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:07.308009] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:07.308025] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:07.308040] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2194751}, scn={val:1736326267147675000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:07.308588] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=3354) [2025-01-08 00:51:07.309802] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29D-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:07.309851] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29D-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326267147675000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326267147675000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:07.310636] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8620, offset=2446010}, end_cursor:ObLogCursor{file_id=1, log_id=8621, offset=2446307}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947a0050, len:3398, log_data_len:297, seq:8620, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:07.310874] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326267147675000, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:07.310906] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:07.310926] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:07.311754] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:07.311862] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=107] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:07.311931] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=37] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326261717746002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326263438968003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.312057] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=123] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:07.312071] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:07.312087] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:07.312103] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:07.312116] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2182108}, scn={val:1736326267189859000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:07.312950] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=650) [2025-01-08 00:51:07.314164] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8621, offset=2446307}, end_cursor:ObLogCursor{file_id=1, log_id=8622, offset=2446604}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947a4050, len:3101, log_data_len:297, seq:8621, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:07.314295] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326267189859000, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:07.314374] INFO [SHARE] add_event (ob_event_history_table_operator.h:266) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1736326267314318), 'checkpoint', 'update_clog_checkpoint', 'tenant_id', 1002, 'ls_id', 1003, 'checkpoint_scn', '{val:1736326267189859000, v:0}', 'service_type', 'MAX_DECIDED_SCN', 'update_clog_checkpoint_times', 120, '', '192.168.159.137', 2882)) [2025-01-08 00:51:07.314396] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:07.314416] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:07.314426] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:07.316551] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46160][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=65] get number(ret=0, service_type_=0, range=1, base_id=1736326267311154000, start_id=1736326267311154000, end_id=1736326267311154001) [2025-01-08 00:51:07.316894] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F3-0-0] [lt=48] get number(ret=0, service_type_=0, range=1, base_id=1736326267311154000, start_id=1736326267311154000, end_id=1736326267311154001) [2025-01-08 00:51:07.319577] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.320297] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA9-0-0] [lt=72] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:07.320344] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CA9-0-0] [lt=43] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:07.320404] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.327880] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=30] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:07.332901] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58404][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:07.333656] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:51:07.347021] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918541E-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:07.347718] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=36] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:1348K/332:279673 disk:1348K/332:279673 t1m2:52K/11:836579 t1m2g0:52K/11:836579 t1002m2:184K/45:576099 t1002m2g0:184K/45:576099 t1001m2:1112K/276:208115 t1001m2g0:1112K/276:208115 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:07.349536] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=90] Slow write(duration=15735, write_len=4096) [2025-01-08 00:51:07.349862] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=322] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8622, offset=2446604}, end_cursor:ObLogCursor{file_id=1, log_id=8623, offset=2446901}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947a8050, len:2804, log_data_len:297, seq:8622, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:07.349947] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=24] current monitor number(seq_=-1) [2025-01-08 00:51:07.350424] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4A-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:07.350470] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4A-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326267132611001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326267132611001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:07.353705] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58404][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38F5-0-0] [lt=62] init point thread id with(&point=0x5572d50e3240, idx_=1060, point=[thread id=58404, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58404) [2025-01-08 00:51:07.362971] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=25] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8623, offset=2446901}, end_cursor:ObLogCursor{file_id=1, log_id=8624, offset=2447198}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947b0050, len:2507, log_data_len:297, seq:8623, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:07.364672] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=51] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8624, offset=2447198}, end_cursor:ObLogCursor{file_id=1, log_id=8625, offset=2447495}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947ac050, len:2210, log_data_len:297, seq:8624, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:07.366880] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17943, miss_cnt:0, cache_read_size:2194751, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:07.370581] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:07.375163] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=123] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:07.396823] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=73] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:07.397911] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=146, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:07.397950] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=38] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=4, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:07.419604] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=16] ====== traversal_flush timer task ====== [2025-01-08 00:51:07.419996] INFO [MDS] flush (mds_table_mgr.cpp:195) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F09-0-0] [lt=51] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326267354573000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279284252000, latest_log_ts:{val:1736326262663505001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.420448] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F09-0-0] [lt=444] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:07.426590] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=67] gc stale ls task succ [2025-01-08 00:51:07.434239] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:07.440509] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:07.442349] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=46] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=629140, clean_num=31457) [2025-01-08 00:51:07.442676] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=66] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:07.442705] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=25] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=131, cache_node_num=131) [2025-01-08 00:51:07.442717] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=131, cache_node_num=131) [2025-01-08 00:51:07.442728] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:07.445364] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=131, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}]) [2025-01-08 00:51:07.445686] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=318] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:07.458384] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=49] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.459290] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.470773] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:07.470879] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:07.470900] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=20] try gc retain ctx [2025-01-08 00:51:07.470908] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:07.470941] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326267132611002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326256925355004, v:0}, keep_alive_scn_:{val:1736326261935253004, v:0}, update_ts_:1736326267470393}) [2025-01-08 00:51:07.471827] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326264665540004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326280087570000, latest_log_ts:{val:1736326263438968001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:07.472034] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=103] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1634, avg_group_log_size=163) [2025-01-08 00:51:07.472055] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:07.472085] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] finish update min start scn(max_decided_scn={val:1736326267132611002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326256925355003, v:0}, keep_alive_scn_:{val:1736326261935253004, v:0}, update_ts_:1736326267470393}) [2025-01-08 00:51:07.472902] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326267132611000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326264665540004, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:07.473038] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] finish update min start scn(max_decided_scn={val:1736326267147675002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326260017888000, v:0}, keep_alive_scn_:{val:1736326264665540004, v:0}, update_ts_:1736326267470393}) [2025-01-08 00:51:07.473311] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=72, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:07.473480] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=64, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:07.473792] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326267147675000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326264665540004, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:07.473858] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=57] finish update min start scn(max_decided_scn={val:1736326267189859002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326258210334002, v:0}, keep_alive_scn_:{val:1736326263623509000, v:0}, update_ts_:1736326267470393}) [2025-01-08 00:51:07.474657] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326267189859000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326264665540004, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:07.475301] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-01-08 00:51:07.475344] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=35] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=69865860, tenant_id=1001) [2025-01-08 00:51:07.475368] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:07.480039] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=46] ====== tenant freeze timer task ====== [2025-01-08 00:51:07.487015] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=31][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:07.487870] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84247262}], scn_array=[cnt:1, {val:1736326267311154001, v:0}], curr_size=122, accum_size=1782, time_cost=14021) [2025-01-08 00:51:07.488006] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43159462}], scn_array=[cnt:1, {val:1736326267354573001, v:0}], curr_size=122, accum_size=7775, time_cost=14184) [2025-01-08 00:51:07.488070] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=92][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2161847}], scn_array=[cnt:1, {val:1736326267132611003, v:0}], curr_size=113, accum_size=235, time_cost=14471) [2025-01-08 00:51:07.490493] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18666112}], scn_array=[cnt:1, {val:1736326267132611003, v:0}], curr_size=113, accum_size=235, time_cost=15310) [2025-01-08 00:51:07.492762] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=27][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:07.519407] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=49] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.541879] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=122] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.544548] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.546373] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=89][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:07.570848] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:07.737780] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4015] clock out of order(cur_ts=1736326267737773, cur_ts_=1736326267578022, delta=159751) [2025-01-08 00:51:07.737894] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] expand thread count(this=0x2ad8f90b0e70, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:07.738160] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1736326267738155, bytes: 0, bw: 0.000000 MB/s, add_ts: 1084185, add_bytes: 0 [2025-01-08 00:51:07.738455] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50378][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] task scheduled too delay(scheduled_delay=230783, expected_run_ts_us_=1736326267507666, ObClockGenerator::getRealClock()=1736326267738451, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05d30, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8efe559b0, time_wheel:0x2ad8cf1f7990, thread_pool:0x2ad8cf1f7170, is_running:1, time_interval:1.00s, expected_run_time:00:51:07.507, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.739501] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46141][pnio1][T0][YB42C0A89F8B-00062B2D17CBB94C-0-0] [lt=24][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=258, fly_ts=119036, send_timestamp=1736326267620454, peer="192.168.159.139:45058", sz=453) [2025-01-08 00:51:07.739722] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F8B-00062B2D1D7BF142-0-0] [lt=22][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=138352, send_timestamp=1736326267601369, peer="192.168.159.139:45074", sz=265) [2025-01-08 00:51:07.740457] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8B-00062B2D183BE82E-0-0] [lt=135][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1808, fly_ts=101694, send_timestamp=1736326267638761, peer="192.168.159.139:45082", sz=257) [2025-01-08 00:51:07.740787] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F8B-00062B2D21BC37E7-0-0] [lt=109][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5377, fly_ts=140854, send_timestamp=1736326267599931, peer="192.168.159.139:45062", sz=418) [2025-01-08 00:51:07.742722] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=150808, expected_run_ts_us_=1736326267591911, ObClockGenerator::getRealClock()=1736326267742720, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe410, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfe6f0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:07.591, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.804958] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE82D-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326267578022000, start_id=1736326267578022001, end_id=1736326267578022002) [2025-01-08 00:51:07.805171] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=34] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:07.805753] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=169] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6718582784, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:07.805883] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37E7-0-0] [lt=28] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=7, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:07.806121] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=37] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=660295680, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=413446144, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:07.806222] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=50] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:07.806325] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=99] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2819, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:07.806423] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=30] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:07.806502] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:07.806528] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:07.806848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50380][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] task scheduled too delay(scheduled_delay=299182, expected_run_ts_us_=1736326267507663, ObClockGenerator::getRealClock()=1736326267806845, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:1, time_interval:100.00ms, expected_run_time:00:51:07.507, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.806986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46270][T1_CoordTR][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] task scheduled too delay(scheduled_delay=200519, expected_run_ts_us_=1736326267606465, ObClockGenerator::getRealClock()=1736326267806985, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3a70, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8950f3d50, time_wheel:0x2ad8879cfd10, thread_pool:0x2ad893be57c0, is_running:1, time_interval:1.00s, expected_run_time:00:51:07.606, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.807343] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=311][errcode=0] task scheduled too delay(scheduled_delay=300878, expected_run_ts_us_=1736326267506463, ObClockGenerator::getRealClock()=1736326267807342, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:07.506, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.809015] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=10398720) [2025-01-08 00:51:07.814427] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:07.814497] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:07.818560] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:07.822844] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=-4016] task executed too delay(delay_time=84305, *p_task_={this:0x2ad8efe05d30, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8efe559b0, time_wheel:0x2ad8cf1f7990, thread_pool:0x2ad8cf1f7170, is_running:0, time_interval:1.00s, expected_run_time:00:51:08.507, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.822917] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50527][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] task executed too delay(delay_time=84215, *p_task_={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:0, time_interval:100.00ms, expected_run_time:00:51:07.749, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.823159] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=35] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:07.823202] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:07.825888] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:07.827351] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:07.827800] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=182] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:07.829562] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2AF-0-0] [lt=27] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326267810235, keepalive_ts:-1}) [2025-01-08 00:51:07.829597] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2AF-0-0] [lt=34] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326267810235, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:07.834436] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=0] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:5, threads_idle_time:7746626632, tenant_id:1001}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:07.835140] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:07.835172] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=99) [2025-01-08 00:51:07.835487] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=35] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:3, threads_idle_time:7746626810, tenant_id:1001}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:07.835557] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:07.835632] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:07.743852] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=91] get number(ret=0, service_type_=0, range=1, base_id=1736326267578022000, start_id=1736326267578022000, end_id=1736326267578022001) [2025-01-08 00:51:07.835728] INFO [STORAGE.TRANS] process_ (ob_gts_rpc.h:219) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=91847] gts local cache not updated(result={tenant_id:1, status:0, srr:[mts=1736326267743843], gts_start:1736326267578022000, gts_end:1736326267578022000}) [2025-01-08 00:51:07.850176] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:51:07.853293] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=46] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:07.853452] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=20] ====== traversal_flush timer task ====== [2025-01-08 00:51:07.853602] INFO [MDS] flush (mds_table_mgr.cpp:195) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E85765-0-0] [lt=22] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326267470393000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279254996000, latest_log_ts:{val:1736326262937841001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:07.853715] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E85765-0-0] [lt=107] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:07.861772] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=36] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1003}, {id:1}, {id:1001}]) [2025-01-08 00:51:07.888859] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=56] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:07.894124] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=197][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18666225}], scn_array=[cnt:1, {val:1736326267132611004, v:0}], curr_size=122, accum_size=357, time_cost=322878) [2025-01-08 00:51:07.896288] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=124][errcode=-4016] task executed too delay(delay_time=153510, *p_task_={this:0x2ad8f0dfe410, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfe6f0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:51:08.919, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.896438] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=-4016] task executed too delay(delay_time=153629, *p_task_={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:07.933, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.896487] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] task executed too delay(delay_time=153673, *p_task_={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:07.952, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.896516] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] task executed too delay(delay_time=68929, *p_task_={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:07.997, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.896532] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] task executed too delay(delay_time=68932, *p_task_={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:08.292, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.896571] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] task executed too delay(delay_time=68967, *p_task_={this:0x2ad8effff190, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff470, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:51:08.270, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.896649] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=-4016] task executed too delay(delay_time=153448, *p_task_={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:0, time_interval:250.00ms, expected_run_time:00:51:07.908, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.896825] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=75][errcode=-4016] task executed too delay(delay_time=90704, *p_task_={this:0x2ad8950ef730, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efa10, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:0, time_interval:500.00ms, expected_run_time:00:51:08.266, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.896893] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] task executed too delay(delay_time=90742, *p_task_={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:0, time_interval:250.00ms, expected_run_time:00:51:08.250, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.897184] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50379][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] task executed too delay(delay_time=90266, *p_task_={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:0, time_interval:100.00ms, expected_run_time:00:51:07.807, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.897529] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4016] task executed too delay(delay_time=90507, *p_task_={this:0x2ad8950f3a70, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8950f3d50, time_wheel:0x2ad8879cfd10, thread_pool:0x2ad893be57c0, is_running:0, time_interval:1.00s, expected_run_time:00:51:08.606, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.897587] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [46271][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=108][errcode=-4016] task executed too delay(delay_time=90190, *p_task_={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:0, time_interval:100.00ms, expected_run_time:00:51:07.906, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:07.898617] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F8A-00062B2D18CA3FBD-0-0] [lt=30][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=205745, send_timestamp=1736326267692869, peer="192.168.159.138:42010", sz=265) [2025-01-08 00:51:07.903580] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F8A-00062B2D208A4547-0-0] [lt=248][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5377, fly_ts=211350, send_timestamp=1736326267692228, peer="192.168.159.138:42000", sz=418) [2025-01-08 00:51:07.968334] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1736326267968330, bytes: 870850174, bw: 0.115668 MB/s, add_ts: 1048873, add_bytes: 127214 [2025-01-08 00:51:07.968607] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.968664] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.968692] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.968731] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:07.968810] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:07.969980] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F2-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326267749409, keepalive_ts:-1}) [2025-01-08 00:51:07.970018] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F2-0-0] [lt=37] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326267749409, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:07.970694] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58385][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF29-0-0] [lt=27] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326267748629, keepalive_ts:-1}) [2025-01-08 00:51:07.970716] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58385][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF29-0-0] [lt=22] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326267748629, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:07.970802] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD5-0-0] [lt=26] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:07.970835] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD5-0-0] [lt=16] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=34) [2025-01-08 00:51:07.974099] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50625][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC39B-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:07.974578] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58389][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC84FC-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=967) [2025-01-08 00:51:07.974695] INFO [LIB] stat (utility.h:1210) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4547-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:07.974767] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58389][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B0-0-0] [lt=27] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326267810493, keepalive_ts:-1}) [2025-01-08 00:51:07.974797] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58389][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B0-0-0] [lt=29] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326267810493, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:08.062207] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5216331705, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:08.063504] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=109] new mysql sessid created(conn.sessid_=3221581808, support_ssl=false) [2025-01-08 00:51:08.063643] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=55] sm conn init succ(conn.sessid_=3221581808, sess.client_addr_="192.168.159.136:39646") [2025-01-08 00:51:08.063671] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x2ad8a0dfc030, session_id:3221581808, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:156}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:08.063745] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8030, session_id:3221580520, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326257119899, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1496, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:08.063785] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221580520, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:51:08.063801] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sock ref clean, do destroy(*s={this:0x2ad8efff8030, session_id:3221580520, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326257119899, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1496, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:08.063861] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] unlock session of tenant(conn.sessid_=3221580520, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:51:08.063878] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221580520, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CC8-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:18, cur_pkt_pos_:24, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:8585484, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:8585485, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:8585486, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:8585486, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:8585486, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:8585486, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:8585486, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:13}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:8585486, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:14}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:8585486, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:15}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{receive:obp20_header_:{payload_len:194, pkt_seq:0, req_id:8585487, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:190, rec_:194, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[11]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:8585487, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:8585487, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[13]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:8585487, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[14]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:8585487, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[15]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:8585487, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[16]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:8585487, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:13}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[17]:{receive:obp20_header_:{payload_len:410, pkt_seq:0, req_id:8585488, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:406, rec_:410, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[18]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:8585488, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[19]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:8585488, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[20]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:8585488, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[21]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:8585488, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:1, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[22]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:8585488, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[23]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:8585488, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:10}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:08.068582] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50379][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=121][errcode=-4016] task executed too delay(delay_time=100550, *p_task_={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:0, time_interval:100.00ms, expected_run_time:00:51:08.766, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.069648] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326256986347, end_time=1736326268069646, dag_status={}) [2025-01-08 00:51:08.069683] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=31] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:08.069701] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:08.069713] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:08.069724] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:08.069732] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:08.069739] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:08.069746] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:08.069753] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:08.069827] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:08.069842] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069868] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069880] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069893] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069904] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069912] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069920] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069928] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069952] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069964] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069977] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069985] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.069994] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070002] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070010] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070032] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070045] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070058] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070069] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070098] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070111] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070124] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070135] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070147] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070170] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070481] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=308] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070501] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070514] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070522] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070530] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070538] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070546] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070554] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070576] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070589] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070602] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070611] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070633] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070646] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070659] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070668] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070676] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070683] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070692] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070714] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070727] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070738] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070749] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070757] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070778] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070791] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070803] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070815] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070824] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070832] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070839] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070862] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070874] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070888] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070896] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070904] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070912] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070931] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070945] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070957] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070969] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070981] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.070993] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.071003] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.071024] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.071036] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.071049] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:08.071062] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:08.071072] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:08.071084] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:08.071089] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:08.071110] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:08.071120] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:08.071129] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:08.071137] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:08.071144] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:08.071150] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:08.071156] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:08.076666] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D38-0-0] [lt=15] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:08.077359] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:08.077899] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6720679936, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:08.078291] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B97-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:08.078443] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B99-0-0] [lt=18] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:08.078723] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4369) [2025-01-08 00:51:08.228906] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D17285296-0-0] [lt=34] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:08.230170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50526][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=280807, expected_run_ts_us_=1736326267949359, ObClockGenerator::getRealClock()=1736326268230168, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfdd30, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8efe09a90, time_wheel:0x2ad8dd57d990, thread_pool:0x2ad8dd57d170, is_running:1, time_interval:1.00s, expected_run_time:00:51:07.949, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.230508] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=51] gc stale ls task succ [2025-01-08 00:51:08.230705] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=68][errcode=0] task scheduled too delay(scheduled_delay=281347, expected_run_ts_us_=1736326267949355, ObClockGenerator::getRealClock()=1736326268230703, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:07.949, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.231625] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] task scheduled too delay(scheduled_delay=325158, expected_run_ts_us_=1736326267906463, ObClockGenerator::getRealClock()=1736326268231623, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:07.906, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.232480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] task scheduled too delay(scheduled_delay=323947, expected_run_ts_us_=1736326267908530, ObClockGenerator::getRealClock()=1736326268232479, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:07.908, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.232768] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:08.232512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50380][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=0] task scheduled too delay(scheduled_delay=224838, expected_run_ts_us_=1736326268007663, ObClockGenerator::getRealClock()=1736326268232502, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:1, time_interval:100.00ms, expected_run_time:00:51:08.766, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.232950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] task scheduled too delay(scheduled_delay=207925, expected_run_ts_us_=1736326268025022, ObClockGenerator::getRealClock()=1736326268232948, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.250, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.235152] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=0] task scheduled too delay(scheduled_delay=301695, expected_run_ts_us_=1736326267933455, ObClockGenerator::getRealClock()=1736326268235151, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:07.933, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.235225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] task scheduled too delay(scheduled_delay=282924, expected_run_ts_us_=1736326267952299, ObClockGenerator::getRealClock()=1736326268235224, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:07.952, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.235369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=131][errcode=0] task scheduled too delay(scheduled_delay=245787, expected_run_ts_us_=1736326267989581, ObClockGenerator::getRealClock()=1736326268235368, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff570, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:07.989, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.235433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=237686, expected_run_ts_us_=1736326267997745, ObClockGenerator::getRealClock()=1736326268235432, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:07.997, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.235496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=230394, expected_run_ts_us_=1736326268005100, ObClockGenerator::getRealClock()=1736326268235495, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfc950, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfcc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:08.510, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.235541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] task scheduled too delay(scheduled_delay=232612, expected_run_ts_us_=1736326268002927, ObClockGenerator::getRealClock()=1736326268235540, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.292, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.235575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] task scheduled too delay(scheduled_delay=143663, expected_run_ts_us_=1736326268091911, ObClockGenerator::getRealClock()=1736326268235574, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe410, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfe6f0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:08.919, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.236125] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [50419][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=7] expand thread count(this=0x2ad8c0eb0e70, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:08.236693] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=27] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:08.236717] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:292) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=24] mysql login queue(mysql_queue->queue_.size()=1) [2025-01-08 00:51:08.236709] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8A-00062B2D1829F4C8-0-0] [lt=22][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1808, fly_ts=333788, send_timestamp=1736326267902919, peer="192.168.159.138:42022", sz=257) [2025-01-08 00:51:08.236793] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A79-0-0] [lt=73] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221581808, conn.client_addr_port_=39646, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221581808, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221581808, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D17784A79-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46334, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523244573683280, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:131, fixup_index:-1, fixup_ash_buffer:{this:0x2ad8d9c65658, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:51:08.239121] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A79-0-0] [lt=164][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:51:08.239288] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A79-0-0] [lt=20] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221581808, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:08.247407] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46141][pnio1][T0][YB42C0A89F8A-00062B2D1D89FE6E-0-0] [lt=40][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=252005, send_timestamp=1736326267995400, peer="192.168.159.138:41998", sz=265) [2025-01-08 00:51:08.249319] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58408][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:08.249372] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74] worker thread created(tenant_->id()=1002, group_id_=1, token=6) [2025-01-08 00:51:08.309405] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58403][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B5A-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4915) [2025-01-08 00:51:08.310583] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:51:08.310670] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=99] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:08.310694] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:51:08.310713] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:08.310733] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:08.310752] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193772204223, flush_io_finished_ret_:0, flush_io_finished_round_:1931, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:08.310850] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=128] recycle one thread(this=0x2ad8885abcf0, total=4, remain=3) [2025-01-08 00:51:08.310882] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=30] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5216331705, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:08.320372] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:08.321393] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:08.321479] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:08.321881] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F4CB-0-0] [lt=50] get number(ret=0, service_type_=0, range=1, base_id=1736326267890517000, start_id=1736326267890517002, end_id=1736326267890517003) [2025-01-08 00:51:08.322754] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAA-0-0] [lt=14] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:08.322788] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAA-0-0] [lt=31] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:08.078746] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:08.326171] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58410][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:08.326571] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:51:08.326634] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:08.329559] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=11] worker thread began to shrink(tenant_->id()=1002, group_id_=18, token=0) [2025-01-08 00:51:08.330751] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=9] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:08.334018] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58147][T1002_L0_G18][T1002][Y0-0000000000000000-0-0] [lt=24] worker thread exit(tenant_->id()=1002, workers_.get_size()=1) [2025-01-08 00:51:08.334108] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58383][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=14] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:51:08.334527] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:08.335696] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:08.335798] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=100] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:08.336492] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:08.337157] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4015] clock out of order(cur_ts=1736326268337155, cur_ts_=1736326267890517, delta=446638) [2025-01-08 00:51:08.337748] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46102][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=25] expand thread count(this=0x5572d527f900, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:08.349709] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918541F-0-0] [lt=30][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:08.350194] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:5, threads_idle_time:15057713656, tenant_id:500}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:08.350645] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:08.350985] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:08.351025] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=35] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:08.351358] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:51:08.351786] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:51:08.352030] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=18] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:08.352685] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=629149696, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:51:08.352726] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=40] === Run print tenant memory usage task === [2025-01-08 00:51:08.352808] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=15] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,874,944 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 631,246,848 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,268,350,025 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 160,591,440 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,040,691,200 max_mem_memstore_can_get_now= 802,957,286 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,268,350,025 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 69,865,860 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 660,295,680 max_mem_memstore_can_get_now= 349,329,398 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,268,350,025 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 240,882,460 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,112,293,376 max_mem_memstore_can_get_now= 1,204,412,368 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:51:08.353053] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=12] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,655,860,224 total_hold= 3,854,565,376 used= 3,475,673,088 freelists_hold= 180,187,136 total_maps= 1,626 total_unmaps= 235 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,874,944 virtual_memory_used= 4,827,381,760 [CHUNK_MGR] 2 MB_CACHE: hold= 77,594,624 count= 37 pushes= 1,815,077 pops= 1,815,040 maps= 1,495 unmaps= 230 [CHUNK_MGR] 4 MB_CACHE: hold= 8,388,608 count= 3 pushes= 6,153 pops= 6,150 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,086 pops= 3,082 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:51:08.353085] INFO print (ob_malloc_time_monitor.cpp:37) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=13] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 5096, delta_count= 4, avg_cost_time= 1274 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= -375968, delta_count= -1, avg_cost_time= 375968 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-01-08 00:51:08.353924] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=143] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:15057717032, tenant_id:500}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:08.354387] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=87] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:08.354548] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=72] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326264299814000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:08.355298] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=19] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:08.355431] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=42] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326265706032000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:08.355465] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=33] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326265706032000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326265706032000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:08.356859] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [50419][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=29] expand thread count(this=0x2ad8c0eb0e70, max_thread_cnt=128, cur_thread_count=5, inc_cnt=1, queue_size=5) [2025-01-08 00:51:08.357928] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=39] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:6, threads_idle_time:7778557112, tenant_id:1002}, cur_thread_count=6, cnt=1, new_thread_count=7) [2025-01-08 00:51:08.358101] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:08.358359] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=32] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:08.359595] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=44] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:3, threads_idle_time:7778557916, tenant_id:1002}, cur_thread_count=6, cnt=1, new_thread_count=7) [2025-01-08 00:51:08.359797] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:08.359829] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:08.360295] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=479) [2025-01-08 00:51:08.360362] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:08.360376] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=120756, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:08.360393] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=91, avg_log_submit_to_flush_time=137798, avg_log_submit_to_slide_time=148620) [2025-01-08 00:51:08.360410] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=19396, log_task={header:{begin_lsn:{lsn:18666225}, end_lsn:{lsn:18666347}, log_id:19396, min_scn:{val:1736326267132611004, v:0}, max_scn:{val:1736326267132611004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18666225}, data_checksum:667309872, accum_checksum:1925601863, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326267571055, freeze_ts:1736326267571064, submit_ts:1736326267571154, flushed_ts:1736326268360284, gen_to_freeze cost time:9, gen_to_submit cost time:99, submit_to_flush cost time:789130}, fs_cb_begin_ts=1736326268360335, log_life_time=789280) [2025-01-08 00:51:08.360468] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=77722, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:08.360574] INFO [SERVER] run (obmp_disconnect.cpp:83) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=81] free session successfully(sessid=3221580520) [2025-01-08 00:51:08.360609] INFO [SERVER] run (obmp_disconnect.cpp:90) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=34] mark session id unused(sessid=3221580520) [2025-01-08 00:51:08.362315] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=146] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=601, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:08.367999] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C93-0-0] [lt=62][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:08.369910] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C93-0-0] [lt=62][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:08.371296] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=33] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:1108K/276:2526111 disk:1108K/276:2526111 t1m2:12K/3:25343996 t1m2g0:12K/3:25343996 t1002m2:56K/14:30716496 t1002m2g0:56K/14:30716496 t1001m2:1040K/259:737409 t1001m2g0:1040K/259:737409 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:08.395073] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=35] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:08.396940] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=58] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:08.397077] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=47] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326265595928000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:08.397160] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:08.397186] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-01-08 00:51:08.397211] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=23] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:08.397230] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:08.397249] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193772310126, flush_io_finished_ret_:0, flush_io_finished_round_:1927, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:08.406855] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:08.406905] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=45] temporary file module use no blocks [2025-01-08 00:51:08.406929] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:08.406945] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:08.406962] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193772320515, flush_io_finished_ret_:0, flush_io_finished_round_:3755, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:08.426730] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=39] gc stale ls task succ [2025-01-08 00:51:08.434742] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:08.434753] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=77, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:08.434821] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:08.434799] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4915, iops:4, bw:21252, [delay/us]:prepare:21, schedule:113, submit:2121, rt:4619, total:6875) [2025-01-08 00:51:08.434852] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=1, iops=1, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:08.434958] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=91] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4915, avg_iops:5, avg_bw:21251, [delay/us]: prepare:21, schedule:113, submit:2121, device:4619, total:6875) [2025-01-08 00:51:08.434987] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:08.435033] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:232, bw:949223, [delay/us]:prepare:1, schedule:439, submit:0, rt:2940, total:3381) [2025-01-08 00:51:08.435059] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:7, bw:35418, [delay/us]:prepare:5, schedule:225, submit:1905, rt:4470, total:6606) [2025-01-08 00:51:08.435070] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1001, ref_cnt=6, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49998, free_result_cnt=49998, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=57, ops=2, iops=59, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:08.435116] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:232, avg_bw:949240, [delay/us]: prepare:1, schedule:439, submit:0, device:2940, total:3381) [2025-01-08 00:51:08.435136] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:5120, avg_iops:7, avg_bw:35419, [delay/us]: prepare:5, schedule:225, submit:1905, device:4470, total:6606) [2025-01-08 00:51:08.435147] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:08.435307] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=140] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:5, bw:21249, [delay/us]:prepare:5, schedule:30, submit:0, rt:3773, total:3811) [2025-01-08 00:51:08.435348] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:3, hold_mem:24576 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:10, bw:38956, [delay/us]:prepare:12, schedule:15411, submit:2671, rt:15557, total:33652) [2025-01-08 00:51:08.435362] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:10624, [delay/us]:prepare:1, schedule:138, submit:496, rt:557, total:1193) [2025-01-08 00:51:08.435371] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=8, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49997, free_result_cnt=49997, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:08.435418] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:6, avg_bw:21249, [delay/us]: prepare:5, schedule:30, submit:0, device:3773, total:3811) [2025-01-08 00:51:08.435438] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:10624, [delay/us]: prepare:1, schedule:138, submit:496, device:557, total:1193) [2025-01-08 00:51:08.435449] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:10, avg_bw:38956, [delay/us]: prepare:12, schedule:15411, submit:2671, device:15557, total:33652) [2025-01-08 00:51:08.435464] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:08.435502] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3541, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:565, total:575) [2025-01-08 00:51:08.435534] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:08.435580] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3541, [delay/us]: prepare:0, schedule:0, submit:600000000, device:565, total:575) [2025-01-08 00:51:08.435603] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:08.435619] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=2, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 2 [2025-01-08 00:51:08.435627] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:08.435689] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:08.435700] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SERVER] net_in=100kB/s, net_out=106kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:08.435838] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:08.435931] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=52] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=3, inc_cnt=1, queue_size=3) [2025-01-08 00:51:08.459570] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:08.537005] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326264399593, last_first_register_time_us=1736324449183670, register_time_us=1736326264399593, curr_time_us=1736326268537001) [2025-01-08 00:51:08.537162] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=119] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326268537126, last_submit_register_req_time_us=1736326268537126, register_parent_reason=1) [2025-01-08 00:51:08.537181] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:08.571433] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=0] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:08.571761] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=16] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6724874240, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:08.572962] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=42] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=55449, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:08.572985] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43160704}], scn_array=[cnt:1, {val:1736326267578022000, v:0}], curr_size=589, accum_size=9484, time_cost=747026) [2025-01-08 00:51:08.573024] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9484) [2025-01-08 00:51:08.573046] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=1254, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:08.573078] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=118092, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:08.573108] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2182474}], scn_array=[cnt:1, {val:1736326267189859004, v:0}], curr_size=122, accum_size=610, time_cost=765272) [2025-01-08 00:51:08.573127] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=610) [2025-01-08 00:51:08.573144] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=656, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:08.573178] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT WRITE LOG COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=154254, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:08.573192] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84247970}], scn_array=[cnt:1, {val:1736326267578022002, v:0}], curr_size=585, accum_size=2953, time_cost=746272) [2025-01-08 00:51:08.573210] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2953) [2025-01-08 00:51:08.573316] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=98] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=1952, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:08.573388] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.137:2882", update_func={old_end_lsn:{lsn:84247970}, new_end_lsn:{lsn:84248555}, old_advance_time_us:1736326267492055, new_ack_time_us:1736326268573384, advance delay(us):1081329}) [2025-01-08 00:51:08.573515] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=92, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:08.573542] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=3905, avg_group_log_size=433) [2025-01-08 00:51:08.573584] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:08.573596] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=163258, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:08.573605] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=4369, avg_log_gen_to_submit_time=4474, avg_log_submit_to_flush_time=157288, avg_log_submit_to_slide_time=158783) [2025-01-08 00:51:08.573618] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=44560, log_task={header:{begin_lsn:{lsn:84247970}, end_lsn:{lsn:84248555}, log_id:44560, min_scn:{val:1736326267578022002, v:0}, max_scn:{val:1736326267578022002, v:0}, data_len:529, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84247970}, data_checksum:2329329928, accum_checksum:1476907323, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326267822561, freeze_ts:1736326267822565, submit_ts:1736326267822595, flushed_ts:1736326268573380, gen_to_freeze cost time:4, gen_to_submit cost time:34, submit_to_flush cost time:750785}, fs_cb_begin_ts=1736326268573563, log_life_time=751002) [2025-01-08 00:51:08.573662] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT FLUSH CB COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=165, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:08.573716] INFO [SERVER] process (obmp_init_db.cpp:53) [46390][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AA9-0-0] [lt=9] init db(db_name=oceanbase) [2025-01-08 00:51:08.574067] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8A-00062B2D1869BB57-0-0] [lt=25][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5392, fly_ts=137744, send_timestamp=1736326268436321, peer="192.168.159.138:42022", sz=256) [2025-01-08 00:51:08.574428] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB57-0-0] [lt=29] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:08.574465] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB57-0-0] [lt=30] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:08.574596] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [46142][pnio1][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-01-08 00:51:08.575462] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] clock out of order(cur_ts=1736326268575460, cur_ts_=1736326268538680, delta=36780) [2025-01-08 00:51:08.575910] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:08.575959] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:08.576000] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:08.576864] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=143406, expected_run_ts_us_=1736326268433455, ObClockGenerator::getRealClock()=1736326268576863, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.433, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.577016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=129][errcode=0] task scheduled too delay(scheduled_delay=124715, expected_run_ts_us_=1736326268452299, ObClockGenerator::getRealClock()=1736326268577015, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.452, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.577102] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=86] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:08.577175] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:08.577392] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=29] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:7749600915, tenant_id:1001}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:08.577430] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:08.577441] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try_wait return eagain(this=0x2ad8f90b0e70, thread=0x2ad91ef05ac0) [2025-01-08 00:51:08.577453] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:7749600915, tenant_id:1001}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:08.577510] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:08.577591] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326267578022002, v:0}, log_ts:{val:1736326267578022002, v:0}, lsn:{lsn:84248026}, submit_ts:1736326267822556}, this:0x2ad89c0ac2d8, is_inited_:true, trans_id:{txid:27911}, ls_id:{id:1}, ctx:0x2ad89c0a88d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=755031, ctx_lock_wait_time=1) [2025-01-08 00:51:08.733697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50378][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=0] task scheduled too delay(scheduled_delay=226024, expected_run_ts_us_=1736326268507666, ObClockGenerator::getRealClock()=1736326268733693, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05d30, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8efe559b0, time_wheel:0x2ad8cf1f7990, thread_pool:0x2ad8cf1f7170, is_running:1, time_interval:1.00s, expected_run_time:00:51:08.507, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.733895] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2953, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:08.733952] INFO [PALF] after_consume (log_io_task.cpp:111) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [io delay](after_consume_ts=1736326268733950, delay_ts=160911) [2025-01-08 00:51:08.733977] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.137:2882", update_func={old_end_lsn:{lsn:43160704}, new_end_lsn:{lsn:43161293}, old_advance_time_us:1736326267491414, new_ack_time_us:1736326268733974, advance delay(us):1242560}) [2025-01-08 00:51:08.734018] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=33] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=6460) [2025-01-08 00:51:08.734121] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=60, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:08.734154] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=8292, avg_group_log_size=691) [2025-01-08 00:51:08.734419] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=245] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=24, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:08.734433] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=78002, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:08.734441] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=1005, avg_log_gen_to_submit_time=1089, avg_log_submit_to_flush_time=67483, avg_log_submit_to_slide_time=76913) [2025-01-08 00:51:08.734451] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=32534, log_task={header:{begin_lsn:{lsn:43160704}, end_lsn:{lsn:43161293}, log_id:32534, min_scn:{val:1736326267578022000, v:0}, max_scn:{val:1736326267578022000, v:0}, data_len:533, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43160704}, data_checksum:1922454847, accum_checksum:1492109558, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326267818471, freeze_ts:1736326267818479, submit_ts:1736326267818538, flushed_ts:1736326268733970, gen_to_freeze cost time:8, gen_to_submit cost time:67, submit_to_flush cost time:915432}, fs_cb_begin_ts=1736326268734408, log_life_time=915937) [2025-01-08 00:51:08.734503] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=93, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:08.740611] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D39-0-0] [lt=9] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:08.742547] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4016] task executed too delay(delay_time=165552, *p_task_={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:08.683, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.742587] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4016] task executed too delay(delay_time=165548, *p_task_={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:08.702, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.742648] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] task executed too delay(delay_time=165594, *p_task_={this:0x2ad8f0dff570, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:51:08.989, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.742730] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=-4016] task executed too delay(delay_time=165669, *p_task_={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:08.747, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.742748] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] task executed too delay(delay_time=165681, *p_task_={this:0x2ad8f0dfc950, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfcc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:51:09.510, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.742779] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] task executed too delay(delay_time=165709, *p_task_={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:08.752, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.742917] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326267311154000, v:0}, new_max_version={val:1736326267311154000, v:0}, last_min_version={val:1736326267045947000, v:0}, last_max_version={val:1736326267045947000, v:0}, total_time_us=922125, delta_us=1431761) [2025-01-08 00:51:08.742955] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326267311154000, v:0}, min_version={val:1736326267311154000, v:0}, max_version={val:1736326267311154000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:08.743036] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:08.750085] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-01-08 00:51:08.750076] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6945] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1736326267470393000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1736326268749992}, version_delta=1279659) [2025-01-08 00:51:08.753699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50374][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] task scheduled too delay(scheduled_delay=247301, expected_run_ts_us_=1736326268506394, ObClockGenerator::getRealClock()=1736326268753697, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe04db0, caller:mds_tenant_service.cpp:mtl_start:125, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8efe05090, time_wheel:0x2ad8cf0e1990, thread_pool:0x2ad8cf0e1170, is_running:1, time_interval:3.00s, expected_run_time:00:51:08.506, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.753920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=47][errcode=0] task scheduled too delay(scheduled_delay=247454, expected_run_ts_us_=1736326268506463, ObClockGenerator::getRealClock()=1736326268753918, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:08.506, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.754107] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50380][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=324][errcode=0] task scheduled too delay(scheduled_delay=246425, expected_run_ts_us_=1736326268507663, ObClockGenerator::getRealClock()=1736326268754089, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:1, time_interval:100.00ms, expected_run_time:00:51:08.507, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.754754] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=16] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:08.755714] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=60] clear dtl interm result cost(us)(clear_cost=5206, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:08.756133] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1736326268756129, bytes: 0, bw: 0.000000 MB/s, add_ts: 1017974, add_bytes: 0 [2025-01-08 00:51:08.756199] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] task scheduled too delay(scheduled_delay=231174, expected_run_ts_us_=1736326268525022, ObClockGenerator::getRealClock()=1736326268756197, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.525, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.757152] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46305][T1_TimerWK1][T1][YB42C0A89F89-00062B2D1768574D-0-0] [lt=33] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.787054533958435059e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=893, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.792507527813269519e+01) [2025-01-08 00:51:08.757678] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326268538680000, start_id=1736326268538680000, end_id=1736326268538680001) [2025-01-08 00:51:08.757866] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=84] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9484, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:08.757961] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B9B-0-0] [lt=13] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:08.758204] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:08.759601] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=17] ====== tenant freeze timer task ====== [2025-01-08 00:51:08.760572] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43161293}, readable_log_tail:{lsn:43161293}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23943475, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:08.761672] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=1097] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43161293}, readable_log_tail:{lsn:43161293}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23943475, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:08.761958] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=284] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:08.762156] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=35] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:08.762701] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326267578022000, v:0}, log_ts:{val:1736326267578022000, v:0}, lsn:{lsn:43160760}, submit_ts:1736326267818463}, this:0x2ad8fb6c4ed8, is_inited_:true, trans_id:{txid:66697}, ls_id:{id:1}, ctx:0x2ad8fb6c14d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=944234, ctx_lock_wait_time=1) [2025-01-08 00:51:08.768656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=176743, expected_run_ts_us_=1736326268591911, ObClockGenerator::getRealClock()=1736326268768655, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe410, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfe6f0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:08.591, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:08.768913] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=108] clear dtl interm result cost(us)(clear_cost=17, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:08.775089] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:08.775673] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58402][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4A-0-0] [lt=178] init point thread id with(&point=0x5572d50e31c0, idx_=1058, point=[thread id=58402, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58402) [2025-01-08 00:51:08.775710] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58402][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4A-0-0] [lt=32] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326268669263, keepalive_ts:-1}) [2025-01-08 00:51:08.775736] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58402][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4A-0-0] [lt=25] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326268669263, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:08.776806] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F89-00062B2D18189D41-0-0] [lt=34][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5387, fly_ts=110592, send_timestamp=1736326268666211, peer="192.168.159.138:42010", sz=479) [2025-01-08 00:51:08.781721] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15059981985, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:08.781752] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:08.781765] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try_wait return eagain(this=0x5572d527f900, thread=0x2ad8acb07ef0) [2025-01-08 00:51:08.781776] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15059981985, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:08.781798] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=20] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7781079005, tenant_id:1002}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:08.781816] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:08.781827] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try_wait return eagain(this=0x2ad8c0eb0e70, thread=0x2ad8d87ff3f0) [2025-01-08 00:51:08.781837] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7781079005, tenant_id:1002}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:08.782646] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58412][TimerWK5][T0][Y0-0000000000000000-0-0] [lt=0] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:08.784416] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:62, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2024173737, recv_hp_rpc_cnt:5131, recv_np_rpc_cnt:103284, recv_lp_rpc_cnt:0, recv_mysql_cnt:9302, recv_task_cnt:3721, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10770227, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=115558 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1832, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:93510, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736326268284924}{group_id:10, queue_size:0, recv_req_cnt:11139, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2071, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326228853642}{group_id:2, queue_size:0, recv_req_cnt:29946, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11853, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:4954, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:26600, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=287 pcode=0x1502:cnt=193 pcode=0x717:cnt=173 pcode=0x710:cnt=66 pcode=0x150b:cnt=60, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:51:08.785313] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=895] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:14703730, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8166, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3563697, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=17, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:51:08.999444] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46308][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=6] expand thread count(this=0x2ad8881fc3b0, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:08.999558] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] clock out of order(cur_ts=1736326268999554, cur_ts_=1736326268733666, delta=265888) [2025-01-08 00:51:09.000171] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=98][errcode=0] task scheduled too delay(scheduled_delay=233809, expected_run_ts_us_=1736326268766349, ObClockGenerator::getRealClock()=1736326269000160, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950ef730, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efa10, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:500.00ms, expected_run_time:00:51:08.766, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.000346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=52][errcode=0] task scheduled too delay(scheduled_delay=225322, expected_run_ts_us_=1736326268775022, ObClockGenerator::getRealClock()=1736326269000345, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.775, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.103610] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:09.104125] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326258677742, end_time=1736326269104121, dag_status={}) [2025-01-08 00:51:09.104170] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=39] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:09.104198] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=27] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:09.104211] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:09.104386] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=174] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:09.104411] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=24] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:09.104423] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:09.104435] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:09.104446] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:09.104466] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:09.104478] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104476] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17] ts response task statistics(alloc_count=59834, free_count=29916) [2025-01-08 00:51:09.104497] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104506] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104515] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104523] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104518] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:09.104546] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104559] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104571] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104579] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104587] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104595] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104618] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104631] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104639] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104648] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104656] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104675] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104689] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104702] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104712] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104729] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104751] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104763] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104775] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104787] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104932] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=144] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104946] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104939] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29] gts statistics(tenant_id=1002, gts_rpc_cnt=9, get_gts_cache_cnt=364, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:09.104970] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104983] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.104996] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105009] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105022] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105030] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105050] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105064] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105076] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105086] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105094] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105102] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105111] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105119] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105127] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105146] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105160] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105173] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105185] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105207] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105220] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105233] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105242] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105249] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105258] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105265] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105289] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105301] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105313] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105572] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=257] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105599] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105612] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105625] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105637] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105647] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105656] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105677] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105691] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105704] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105715] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105724] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105732] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105740] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105748] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:09.105784] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:09.105793] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:09.105801] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:09.105806] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:09.105812] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:09.105817] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:09.105835] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:09.105843] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:09.105851] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:09.105860] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:09.105867] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:09.106852] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4016] task executed too delay(delay_time=106617, *p_task_={this:0x2ad8950ef730, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efa10, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:0, time_interval:500.00ms, expected_run_time:00:51:09.266, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.107007] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-4016] task executed too delay(delay_time=106617, *p_task_={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:0, time_interval:250.00ms, expected_run_time:00:51:09.250, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.108124] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:09.108475] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=336] ====== check clog disk timer task ====== [2025-01-08 00:51:09.108490] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:09.108820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] task scheduled too delay(scheduled_delay=259461, expected_run_ts_us_=1736326268849355, ObClockGenerator::getRealClock()=1736326269108817, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:08.849, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.109299] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:09.109422] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=22] BlockGCTimerTask success(ret=0, cost_time_us=17, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:0}}) [2025-01-08 00:51:09.109837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46270][T1_CoordTR][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] task scheduled too delay(scheduled_delay=271663, expected_run_ts_us_=1736326268838171, ObClockGenerator::getRealClock()=1736326269109835, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f32b0, caller:ob_leader_coordinator.cpp:mtl_start:104, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8950f3590, time_wheel:0x2ad8879cfd10, thread_pool:0x2ad893be57c0, is_running:1, time_interval:5.00s, expected_run_time:00:51:08.838, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.252659] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.252744] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.252789] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.252829] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.253386] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:09.255920] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A74-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326269255901}) [2025-01-08 00:51:09.260912] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}, {id:1003}]) [2025-01-08 00:51:09.261039] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326267132611003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=2128411, in_cluster_service=true, cluster_version={val:1736326267132611002, v:0}, min_cluster_version={val:1736326267132611002, v:0}, max_cluster_version={val:1736326267132611002, v:0}, get_cluster_version_err=0, cluster_version_delta=2128411, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18245, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=497178549, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:09.262370] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=29] gc stale ls task succ [2025-01-08 00:51:09.264359] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=8838) [2025-01-08 00:51:09.264398] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:09.264552] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:09.264769] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285967-0-0] [lt=27] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326269264697, sql_renew_time:1736326269264697, is_mark_fail:false}]) [2025-01-08 00:51:09.264815] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285967-0-0] [lt=45] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326269255901}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326269264697, sql_renew_time:1736326269264697, is_mark_fail:false}]) [2025-01-08 00:51:09.269993] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=28] expand thread count(this=0x2ad8f90b0e70, max_thread_cnt=128, cur_thread_count=6, inc_cnt=1, queue_size=6) [2025-01-08 00:51:09.270418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] task scheduled too delay(scheduled_delay=245392, expected_run_ts_us_=1736326269025022, ObClockGenerator::getRealClock()=1736326269270415, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:09.250, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.270692] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [58409][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-01-08 00:51:09.270728] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [58409][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:09.270785] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [58409][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=16] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:09.274493] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=25] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:09.274537] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] task executed too delay(delay_time=169052, *p_task_={this:0x2ad8f0dfdd30, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8efe09a90, time_wheel:0x2ad8dd57d990, thread_pool:0x2ad8dd57d170, is_running:0, time_interval:1.00s, expected_run_time:00:51:09.949, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.275079] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6710194176, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:09.278860] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:09.279372] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:09.279642] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50380][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] task scheduled too delay(scheduled_delay=471975, expected_run_ts_us_=1736326268807663, ObClockGenerator::getRealClock()=1736326269279639, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:1, time_interval:100.00ms, expected_run_time:00:51:08.807, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.280193] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=51][errcode=0] task scheduled too delay(scheduled_delay=473726, expected_run_ts_us_=1736326268806463, ObClockGenerator::getRealClock()=1736326269280190, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:08.806, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.281325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:09.275, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.281457] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=31] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:09.288002] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:09.288099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] task scheduled too delay(scheduled_delay=405571, expected_run_ts_us_=1736326268882524, ObClockGenerator::getRealClock()=1736326269288097, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8a0dff190, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff470, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:1, time_interval:500.00ms, expected_run_time:00:51:08.882, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.288168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] task scheduled too delay(scheduled_delay=379637, expected_run_ts_us_=1736326268908530, ObClockGenerator::getRealClock()=1736326269288168, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.908, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.288375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:09.149, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.288904] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:09.290273] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=70] LogLoopThread round_cost_time(us)(round_cost_time=24) [2025-01-08 00:51:09.290443] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2166056, this={dio_aligned_buf:{buf_write_offset:3490, buf_padding_size:606, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:22566, truncate_used_ts:5080}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2162688, buf_len=122, total_write_size=11684, total_write_size_after_dio=405504, ob_pwrite_used_ts=2246866, count=96, each_pwrite_cost=23404) [2025-01-08 00:51:09.290491] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=227105, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:09.290507] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2161960}], scn_array=[cnt:1, {val:1736326267132611004, v:0}], curr_size=122, accum_size=357, time_cost=1552652) [2025-01-08 00:51:09.290529] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=357) [2025-01-08 00:51:09.290550] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=25193, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:09.290593] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="192.168.159.137:2882", server="192.168.159.137:2882", update_func={old_end_lsn:{lsn:2161960}, new_end_lsn:{lsn:2162082}, old_advance_time_us:1736326267488365, new_ack_time_us:1736326269290588, advance delay(us):1802223}) [2025-01-08 00:51:09.290627] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=122) [2025-01-08 00:51:09.290668] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:09.290686] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=268843, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:09.290701] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=17763, log_task={header:{begin_lsn:{lsn:2161960}, end_lsn:{lsn:2162082}, log_id:17763, min_scn:{val:1736326267132611004, v:0}, max_scn:{val:1736326267132611004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2161960}, data_checksum:2222604892, accum_checksum:3669121729, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326267571179, freeze_ts:1736326267571182, submit_ts:1736326267571210, flushed_ts:1736326269290583, gen_to_freeze cost time:3, gen_to_submit cost time:31, submit_to_flush cost time:1719373}, fs_cb_begin_ts=1736326269290651, log_life_time=1719472) [2025-01-08 00:51:09.290753] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=28, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:09.290796] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:948) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] cb cost too much time(lsn={lsn:2162016}, scn={val:1736326267132611004, v:0}, idx=28, total_cost_time=1703415, append_cost_time=39, cb_wait_thread_time=1719574, cb_wait_commit_time=0, cb_cost_time=-16198, append_start_time=1736326267571177, append_finish_time=1736326267571216, cb_first_handle_time=1736326269290790, cb_finish_time=1736326269274592) [2025-01-08 00:51:09.290831] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=235, hit_cnt=2, read_cnt=2, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:09.291854] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT WRITE LOG COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=440646, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:09.291892] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2195117}], scn_array=[cnt:1, {val:1736326267147675004, v:0}], curr_size=122, accum_size=366, time_cost=1316995) [2025-01-08 00:51:09.291921] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=366) [2025-01-08 00:51:09.292022] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=109, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:09.291932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=65][errcode=0] task scheduled too delay(scheduled_delay=358474, expected_run_ts_us_=1736326268933455, ObClockGenerator::getRealClock()=1736326269291930, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.933, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.292114] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=156][errcode=0] task scheduled too delay(scheduled_delay=339813, expected_run_ts_us_=1736326268952299, ObClockGenerator::getRealClock()=1736326269292113, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.952, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.292734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=303151, expected_run_ts_us_=1736326268989581, ObClockGenerator::getRealClock()=1736326269292733, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff570, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:08.989, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.292784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] task scheduled too delay(scheduled_delay=295038, expected_run_ts_us_=1736326268997745, ObClockGenerator::getRealClock()=1736326269292783, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:08.997, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.292830] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] task scheduled too delay(scheduled_delay=287729, expected_run_ts_us_=1736326269005100, ObClockGenerator::getRealClock()=1736326269292830, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfc950, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfcc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:09.510, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.292864] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=289936, expected_run_ts_us_=1736326269002927, ObClockGenerator::getRealClock()=1736326269292864, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:09.292, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.292883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=200972, expected_run_ts_us_=1736326269091911, ObClockGenerator::getRealClock()=1736326269292883, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe410, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfe6f0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:09.919, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.293007] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=46, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:09.294933] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50527][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] task executed too delay(delay_time=186067, *p_task_={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:0, time_interval:100.00ms, expected_run_time:00:51:09.349, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:09.295705] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=45] LogLoopThread round_cost_time(us)(round_cost_time=693) [2025-01-08 00:51:09.295747] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:44562, max_lsn:84249222, max_scn:{val:1736326268538680000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84248555}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42306182}}, last_submit_lsn:{lsn:84248555}, last_submit_end_lsn:{lsn:84248677}, last_submit_log_id:44561, last_submit_log_pid:1, max_flushed_lsn:{lsn:84247970}, max_flushed_end_lsn:{lsn:84248555}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84248555}, last_slide_log_id:44560, last_slide_scn:{val:1736326267578022002, v:0}, last_slide_lsn:{lsn:84247970}, last_slide_end_lsn:{lsn:84248555}, last_slide_log_pid:1, last_slide_log_accum_checksum:1476907323, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:44561, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84248555}, readable_log_tail:{lsn:84248555}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49960981, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:51:09.297098] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1342] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326268573384, last_flushed_end_lsn:{lsn:84248555}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326267493448, last_flushed_end_lsn:{lsn:84247970}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326268309385, last_flushed_end_lsn:{lsn:84248555}}]) [2025-01-08 00:51:09.297123] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=44561, log_task={header:{begin_lsn:{lsn:84248555}, end_lsn:{lsn:84248677}, log_id:44561, min_scn:{val:1736326267578022003, v:0}, max_scn:{val:1736326267578022003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84248555}, data_checksum:667309872, accum_checksum:704720349, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1736326268078709, freeze_ts:1736326268573437, submit_ts:1736326268573531, flushed_ts:-1, gen_to_freeze cost time:494728, gen_to_submit cost time:494822, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326268573384, last_flushed_end_lsn:{lsn:84248555}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326267493448, last_flushed_end_lsn:{lsn:84247970}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326268309385, last_flushed_end_lsn:{lsn:84248555}}]) [2025-01-08 00:51:09.309337] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=9] finish to resize log external storage handler(current_ts=1736326269309205, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:09.309391] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=53] snapshot_traffic_info once [2025-01-08 00:51:09.309980] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=33] blacklist_loop exec finished(cost_time=44, is_enabled=true, send_cnt=0) [2025-01-08 00:51:09.310825] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=29] gc stale ls task succ [2025-01-08 00:51:09.311121] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=41] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:09.312694] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=96] temporary file module use no blocks [2025-01-08 00:51:09.312718] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:09.312743] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:09.312783] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=38] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193773224639, flush_io_finished_ret_:0, flush_io_finished_round_:1932, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:09.315212] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [46142][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO resp cb not found: packet_id=622061 [2025-01-08 00:51:09.316051] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F8A-00062B2D1829F4CE-0-0] [lt=109][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1808, fly_ts=217668, send_timestamp=1736326269098380, peer="192.168.159.138:42000", sz=257) [2025-01-08 00:51:09.316696] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58410][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B5B-0-0] [lt=194] init point thread id with(&point=0x5572d50e33c0, idx_=1066, point=[thread id=58410, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58410) [2025-01-08 00:51:09.316793] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58410][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B5B-0-0] [lt=81][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:84248555}, new_end_lsn:{lsn:84248677}, old_advance_time_us:1736326268309385, new_ack_time_us:1736326269316788, advance delay(us):1007403}) [2025-01-08 00:51:09.316835] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58410][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B5B-0-0] [lt=33] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=585) [2025-01-08 00:51:09.317081] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58410][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA67F3-0-0] [lt=16][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.138:2882", update_func={old_end_lsn:{lsn:84247970}, new_end_lsn:{lsn:84248677}, old_advance_time_us:1736326267493448, new_ack_time_us:1736326269317078, advance delay(us):1823630}) [2025-01-08 00:51:09.318047] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA38F9-0-0] [lt=15][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.138:2882", update_func={old_end_lsn:{lsn:43160704}, new_end_lsn:{lsn:43161293}, old_advance_time_us:1736326267492261, new_ack_time_us:1736326269318043, advance delay(us):1825782}) [2025-01-08 00:51:09.318257] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4549-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:09.318323] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4549-0-0] [lt=34] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=4, avg_log_batch_cnt=0, total_group_log_size=488, avg_group_log_size=122) [2025-01-08 00:51:09.318352] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4549-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=854) [2025-01-08 00:51:09.318407] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4549-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=17, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:09.318429] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4549-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=239036, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:09.318444] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4549-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=59, avg_log_submit_to_flush_time=150955, avg_log_submit_to_slide_time=187828) [2025-01-08 00:51:09.318462] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4549-0-0] [lt=17][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18035, log_task={header:{begin_lsn:{lsn:2195117}, end_lsn:{lsn:2195239}, log_id:18035, min_scn:{val:1736326267147675004, v:0}, max_scn:{val:1736326267147675004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2194995}, prev_proposal_id:1, committed_end_lsn:{lsn:2195117}, data_checksum:1715074492, accum_checksum:2190929293, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326267974685, freeze_ts:1736326267974685, submit_ts:1736326267974723, flushed_ts:1736326269292975, gen_to_freeze cost time:0, gen_to_submit cost time:38, submit_to_flush cost time:1318252}, fs_cb_begin_ts=1736326269318374, log_life_time=1343689) [2025-01-08 00:51:09.318819] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F8A-00062B2D1829F4CC-0-0] [lt=35][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1808, fly_ts=220819, send_timestamp=1736326269097998, peer="192.168.159.138:42010", sz=256) [2025-01-08 00:51:09.320169] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.320267] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58408][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9133-0-0] [lt=22] init point thread id with(&point=0x5572d50e3340, idx_=1064, point=[thread id=58408, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58408) [2025-01-08 00:51:09.320312] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58408][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9133-0-0] [lt=37][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="192.168.159.137:2882", server="192.168.159.138:2882", update_func={old_end_lsn:{lsn:2161960}, new_end_lsn:{lsn:2162082}, old_advance_time_us:1736326267490250, new_ack_time_us:1736326269320310, advance delay(us):1830060}) [2025-01-08 00:51:09.320486] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58408][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9135-0-0] [lt=21][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.138:2882", update_func={old_end_lsn:{lsn:18666225}, new_end_lsn:{lsn:18666347}, old_advance_time_us:1736326267486542, new_ack_time_us:1736326269320483, advance delay(us):1833941}) [2025-01-08 00:51:09.321009] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D035-0-0] [lt=51][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.322836] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=854, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:09.322873] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17947, miss_cnt:0, cache_read_size:2195239, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:09.323119] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.323221] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=12] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=113, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:09.323238] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=17] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=3, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:09.323848] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:194) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] async_cb handler cost too much time(total_time=573770, decode_time=-1736326268750076, process_time=0, session_destroy_time=1736326269323846, ret=0, pcode=1861) [2025-01-08 00:51:09.323875] WDIAG stat_cleanup (debug.c:35) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] PNIO [delay_warn] cost too much time: 573552us, procedure: pn_pktc_resp_cb [2025-01-08 00:51:09.323893] WDIAG pktc_handle_req_queue (pktc_post.h:138) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] PNIO [delay_warn] delay high: 786385 [2025-01-08 00:51:09.323905] WDIAG eloop_delay_warn (debug.h:93) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO [delay_warn] eloop handle events delay high: 573552, malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=573552/1 [2025-01-08 00:51:09.323913] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO [ratelimit] time: 1736326269323912, bytes: 870917230, bw: 0.047175 MB/s, add_ts: 1355582, add_bytes: 67056 [2025-01-08 00:51:09.324035] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46141][pnio1][T0][YB42C0A89F89-00062B2D21F89BDF-0-0] [lt=9][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5387, fly_ts=657041, send_timestamp=1736326268666993, peer="192.168.159.138:41998", sz=480) [2025-01-08 00:51:09.324918] WDIAG [RPC] wait (ob_async_rpc_proxy.h:453) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] execute rpc failed(rc=-4012, server="192.168.159.138:2882", timeout=200000, packet code=1861, arg={tenant_id:1002, tenant_info:{tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326266130647000, replayable_scn:1736326266130647000, readable_scn:1736326265815330002, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}, ora_rowscn:1736326267320455000, finish_data_version:0, data_version_barrier_scn:{val:0, v:0}}, ret=0) [2025-01-08 00:51:09.324955] WDIAG [RPC] wait (ob_async_rpc_proxy.h:453) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] execute rpc failed(rc=-4012, server="192.168.159.139:2882", timeout=200000, packet code=1861, arg={tenant_id:1002, tenant_info:{tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326266130647000, replayable_scn:1736326266130647000, readable_scn:1736326265815330002, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}, ora_rowscn:1736326267320455000, finish_data_version:0, data_version_barrier_scn:{val:0, v:0}}, ret=0) [2025-01-08 00:51:09.326788] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87070-0-0] [lt=11][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.326839] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:09.326856] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:09.333708] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAB-0-0] [lt=26] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:09.333767] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAB-0-0] [lt=54] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:09.337256] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:342) [46499][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB845-0-0] [lt=32][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:27912}, ref_cnt:1, state:"RUNNING", commit_version:{val:1736326268538680000, v:0}, start_scn:{val:1736326268538680000, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout_ts:1736326299223746, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326268733666000, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, data_trans_id_:{txid:27912}, data_sql_sequence_:{branch:0, seq:3}, read_latest_:false, read_uncommitted_:false, scn_:{val:1736326268538680000, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1736326268538680000, v:0}}) [2025-01-08 00:51:09.345248] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46497][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB8A-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774259081, cache_obj->added_lc()=false, cache_obj->get_object_id()=13595, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.345151] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46383][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1949CA37-0-0] [lt=32][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774258984, cache_obj->added_lc()=false, cache_obj->get_object_id()=13596, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.347470] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58213][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB5F-0-0] [lt=38] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:09.347503] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58213][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB5F-0-0] [lt=26] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:09.348568] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:09.348594] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:09.349544] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8B-00062B2D187BAEBF-0-0] [lt=18][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5392, fly_ts=775823, send_timestamp=1736326268573718, peer="192.168.159.139:45082", sz=256) [2025-01-08 00:51:09.349877] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87073-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.349918] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:09.349941] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:09.350247] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=57] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:09.350280] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=30] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:09.350292] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:09.350301] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:09.355740] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:09.356692] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=947] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:09.358080] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D16C84A74-0-0] [lt=76] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:09.358173] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185420-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.358414] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=16] current monitor number(seq_=-1) [2025-01-08 00:51:09.363567] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58389][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9136-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=0) [2025-01-08 00:51:09.363654] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:09.364369] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEBF-0-0] [lt=30] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:09.364398] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEBF-0-0] [lt=22] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:09.364928] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=44] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:09.386411] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:09.386487] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=74] try gc retain ctx [2025-01-08 00:51:09.386499] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-01-08 00:51:09.386535] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] get_max_applied_scn(scn={val:1736326267470393001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43161293}, palf_committed_end_scn_:{val:1736326267578022000, v:0}, last_check_scn_:{val:1736326268733666001, v:0}, max_applied_cb_scn_:{val:1736326267470393001, v:0}}) [2025-01-08 00:51:09.386567] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] finish update min start scn(max_decided_scn={val:1736326267470393001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326258743968001, v:0}, keep_alive_scn_:{val:1736326263763810001, v:0}, update_ts_:1736326269366594}) [2025-01-08 00:51:09.387290] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279284252000, latest_log_ts:{val:1736326262663505001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:09.387691] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=247111] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:09.387710] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] try gc retain ctx [2025-01-08 00:51:09.387716] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] try to retry start_working [2025-01-08 00:51:09.387746] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1736326267470393000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84248555}, palf_committed_end_scn_:{val:1736326267578022002, v:0}, last_check_scn_:{val:1736326267578022002, v:0}, max_applied_cb_scn_:{val:1736326267470393000, v:0}}) [2025-01-08 00:51:09.387772] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84248555}, readable_log_tail:{lsn:84248555}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49960981, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:09.387847] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=73] finish update min start scn(max_decided_scn={val:1736326267470393000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326258720921001, v:0}, keep_alive_scn_:{val:1736326263731517001, v:0}, update_ts_:1736326269366594}) [2025-01-08 00:51:09.388609] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279254996000, latest_log_ts:{val:1736326262937841001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:09.389385] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=85, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:09.389418] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1089, avg_group_log_size=121) [2025-01-08 00:51:09.389436] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1089) [2025-01-08 00:51:09.389449] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:09.389565] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=73][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:09.406316] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDD3-0-0] [lt=36][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774320148, cache_obj->added_lc()=false, cache_obj->get_object_id()=13597, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.410491] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56] get number(ret=0, service_type_=0, range=1, base_id=1736326269366594000, start_id=1736326269366594000, end_id=1736326269366594001) [2025-01-08 00:51:09.410572] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=611) [2025-01-08 00:51:09.410591] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:09.415624] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46496][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C61F-0-0] [lt=5][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774329458, cache_obj->added_lc()=false, cache_obj->get_object_id()=13594, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.432939] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=26] gc stale ls task succ [2025-01-08 00:51:09.440803] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46497][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1949CA39-0-0] [lt=19][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774354635, cache_obj->added_lc()=false, cache_obj->get_object_id()=13598, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.454082] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221581910, support_ssl=false) [2025-01-08 00:51:09.454232] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221581910, sess.client_addr_="192.168.159.136:39650") [2025-01-08 00:51:09.454256] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x2ad8a0de8030, session_id:3221581910, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:183}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:09.454997] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] clock out of order(cur_ts=1736326269454995, cur_ts_=1736326269366594, delta=88401) [2025-01-08 00:51:09.461951] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.461996] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.471856] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:09.471958] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7A-0-0] [lt=47] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221581910, conn.client_addr_port_=39650, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221581910, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221581910, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D17784A7A-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46334, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523247911197359, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:146, fixup_index:-1, fixup_ash_buffer:{this:0x2ad8c202b658, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:51:09.474032] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7A-0-0] [lt=345][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:51:09.474101] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7A-0-0] [lt=20] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221581910, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:09.477808] WDIAG [PALF] is_prev_log_pid_match (log_sliding_window.cpp:3913) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A454D-0-0] [lt=25][errcode=-4018] get_prev_log_info_ failed(tmp_ret=-4018, log_id=18038, palf_id=1002, self="192.168.159.137:2882", log_id=18038, lsn={lsn:2195483}, prev_lsn={lsn:2195361}, prev_log_pid=1) [2025-01-08 00:51:09.477922] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8501-0-0] [lt=62][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:2162082}, new_end_lsn:{lsn:2162204}, old_advance_time_us:1736326267974569, new_ack_time_us:1736326269477918, advance delay(us):1503349}) [2025-01-08 00:51:09.488261] INFO [SERVER] process (obmp_init_db.cpp:53) [46388][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B7B-0-0] [lt=20] init db(db_name=oceanbase) [2025-01-08 00:51:09.489808] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] success to update min reserved snapshot(reserved_snapshot=1736324465595928000, duration=1800, snapshot_gc_ts=1736326265595928000) [2025-01-08 00:51:09.489940] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=74] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=357) [2025-01-08 00:51:09.489968] INFO [LIB] stat (utility.h:1210) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:09.490554] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=31] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-01-08 00:51:09.494817] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:09.494922] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=58] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:1, bw:3863, [delay/us]:prepare:7, schedule:75715, submit:65868, rt:604031, total:745621) [2025-01-08 00:51:09.494951] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:09.495118] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=143] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:1, avg_bw:3863, [delay/us]: prepare:7, schedule:75715, submit:65868, device:604031, total:745621) [2025-01-08 00:51:09.495143] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:09.495194] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:2, bw:7727, [delay/us]:prepare:3, schedule:8971, submit:2, rt:213032, total:222008) [2025-01-08 00:51:09.495433] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=218] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:1, bw:3863, [delay/us]:prepare:6, schedule:79198, submit:63057, rt:603956, total:746217) [2025-01-08 00:51:09.495453] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1001, ref_cnt=6, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49998, free_result_cnt=49998, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:09.495518] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=54] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:2, avg_bw:7725, [delay/us]: prepare:3, schedule:8971, submit:2, device:213032, total:222008) [2025-01-08 00:51:09.495553] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:1, avg_bw:3862, [delay/us]: prepare:6, schedule:79198, submit:63057, device:603956, total:746217) [2025-01-08 00:51:09.495579] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:09.495603] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=39] ====== tenant freeze timer task ====== [2025-01-08 00:51:09.495658] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:3, hold_mem:24576 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:11588, [delay/us]:prepare:9, schedule:87380, submit:311361, rt:680068, total:1078819) [2025-01-08 00:51:09.495686] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS TENANT](tenant_id=1002, ref_cnt=10, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31098816, free_req_cnt=49996, free_result_cnt=49996, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:09.495775] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=66] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:3, avg_bw:11588, [delay/us]: prepare:9, schedule:87380, submit:311361, device:680068, total:1078819) [2025-01-08 00:51:09.495800] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:09.495844] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3862, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:319543, total:319551) [2025-01-08 00:51:09.496169] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=301] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:09.496223] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3861, [delay/us]: prepare:0, schedule:0, submit:600000000, device:319543, total:319551) [2025-01-08 00:51:09.496251] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:09.496280] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:09.496377] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=90] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:09.496395] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:09.496406] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=104kB/s, net_out=80kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:09.496999] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:09.497045] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:09.498682] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8503-0-0] [lt=53][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:18666347}, new_end_lsn:{lsn:18666469}, old_advance_time_us:1736326267974640, new_ack_time_us:1736326269498677, advance delay(us):1524037}) [2025-01-08 00:51:09.500448] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=46] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1195328, replace_num=15728) [2025-01-08 00:51:09.500476] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:09.502404] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=117] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6718582784, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:09.502961] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46386][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B7E-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12358, add_timestamp:1736326269502894}) [2025-01-08 00:51:09.503078] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46386][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B7E-0-0] [lt=79] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12396, add_timestamp:1736326269503072}) [2025-01-08 00:51:09.503083] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285968-0-0] [lt=49] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12358, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326269503019, sql_renew_time:1736326269503019, is_mark_fail:false}]) [2025-01-08 00:51:09.503244] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285968-0-0] [lt=159] success to process renew task(task={tenant_id:1, table_id:12358, add_timestamp:1736326269502894}, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326269503019, sql_renew_time:1736326269503019, is_mark_fail:false}]) [2025-01-08 00:51:09.503407] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285969-0-0] [lt=141] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12396, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326269503397, sql_renew_time:1736326269503397, is_mark_fail:false}]) [2025-01-08 00:51:09.503426] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285969-0-0] [lt=18] success to process renew task(task={tenant_id:1, table_id:12396, add_timestamp:1736326269503072}, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326269503397, sql_renew_time:1736326269503397, is_mark_fail:false}]) [2025-01-08 00:51:09.503999] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F8B-00062B2D183BE833-0-0] [lt=65][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1808, fly_ts=246572, send_timestamp=1736326269257425, peer="192.168.159.139:45074", sz=256) [2025-01-08 00:51:09.505466] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58408][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9134-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=854) [2025-01-08 00:51:09.505542] INFO [LIB] stat (utility.h:1210) [58408][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9134-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=30, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:09.505567] INFO [LIB] stat (utility.h:1210) [58408][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9134-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=286896, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:09.505582] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58408][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9134-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=22, avg_log_submit_to_flush_time=157214, avg_log_submit_to_slide_time=372715) [2025-01-08 00:51:09.505601] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58408][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9134-0-0] [lt=19][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17931, log_task={header:{begin_lsn:{lsn:2182474}, end_lsn:{lsn:2182596}, log_id:17931, min_scn:{val:1736326267189859004, v:0}, max_scn:{val:1736326267189859004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2182352}, prev_proposal_id:1, committed_end_lsn:{lsn:2182474}, data_checksum:940009244, accum_checksum:2697953353, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326267805866, freeze_ts:1736326267805866, submit_ts:1736326267805912, flushed_ts:1736326268577137, gen_to_freeze cost time:0, gen_to_submit cost time:46, submit_to_flush cost time:771225}, fs_cb_begin_ts=1736326269505501, log_life_time=1699635) [2025-01-08 00:51:09.506025] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58385][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2C62-0-0] [lt=36][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:43161293}, new_end_lsn:{lsn:43161947}, old_advance_time_us:1736326268345906, new_ack_time_us:1736326269506021, advance delay(us):1160115}) [2025-01-08 00:51:09.506158] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58213][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC7-0-0] [lt=82] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:09.506305] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58213][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC7-0-0] [lt=23] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:09.507981] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=610, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:09.508012] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17842, miss_cnt:0, cache_read_size:2182596, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:09.509693] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4B-0-0] [lt=48] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326269580345, keepalive_ts:-1}) [2025-01-08 00:51:09.509747] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4B-0-0] [lt=53] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326269580345, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:09.505783] INFO [LIB] stat (utility.h:1210) [58408][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37E9-0-0] [lt=41] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:09.510008] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58408][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37E9-0-0] [lt=4205] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=845, avg_group_log_size=120) [2025-01-08 00:51:09.523648] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46386][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B7E-0-0] [lt=20][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.524019] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.526216] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B7F-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11001, add_timestamp:1736326269526028}) [2025-01-08 00:51:09.530840] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46494][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1949CA3B-0-0] [lt=15][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774444674, cache_obj->added_lc()=false, cache_obj->get_object_id()=13599, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.534441] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC39E-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.535623] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B7F-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.536072] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:09.536109] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=69865860, tenant_id=1001) [2025-01-08 00:51:09.536141] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:09.537608] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596A-0-0] [lt=13] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11001, vtable_type={tenant_id:1, type:2}) [46232]OB_MALLOC COST TOO MUCH TIME, cost_time=1104682, tenant_id=500, label=CoStack, ctx_id=8, prio=1, size=515136, owner: ObMalloc, click_count: 1, time dist:[ALLOC_CHUNK_END=1104679], seq:[0] [2025-01-08 00:51:09.547360] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C94-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.548631] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46506][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C622-0-0] [lt=50] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.784654479026794434e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=892, Throttle Trigger(MB)=535, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.807517616677767691e+01) [2025-01-08 00:51:09.549425] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C94-0-0] [lt=58][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.551785] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.551889] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.554431] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596A-0-0] [lt=26] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11001, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326269554361, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:09.554472] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596A-0-0] [lt=40] success to process renew task(task={tenant_id:1, table_id:11001, add_timestamp:1736326269526028}, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326269554361, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:09.557935] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=29] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:09.558603] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C624-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1736326269558308000, start_id=1736326269558308000, end_id=1736326269558308001) [2025-01-08 00:51:09.558630] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:948) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=55][errcode=-4389] cb cost too much time(lsn={lsn:84248026}, scn={val:1736326267578022002, v:0}, idx=18, total_cost_time=1735751, append_cost_time=62, cb_wait_thread_time=754967, cb_wait_commit_time=0, cb_cost_time=980722, append_start_time=1736326267822557, append_finish_time=1736326267822619, cb_first_handle_time=1736326268577586, cb_finish_time=1736326269558308) [2025-01-08 00:51:08.787009] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1694] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:13761342, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3550517, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:51:09.560683] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=773554] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:21, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:869629154, recv_hp_rpc_cnt:3702, recv_np_rpc_cnt:37638, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:772288, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=65077 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:67326, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736326268326632}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2169, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326218958545}{group_id:2, queue_size:0, recv_req_cnt:15346, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18398, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=284 pcode=0x51c:cnt=234 pcode=0x717:cnt=181 pcode=0x710:cnt=91 pcode=0x150b:cnt=64, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:51:09.562490] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1805] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:57, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9409903648, recv_hp_rpc_cnt:77, recv_np_rpc_cnt:34637, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2920538, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=24864 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:1, recv_req_cnt:6956, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326269316514}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:149034, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:6, nesting_worker_cnt:0, token_change:1736326267750931}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326242277705}{group_id:19, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326264650105}{group_id:2, queue_size:0, recv_req_cnt:36793, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:11000, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=354 pcode=0x717:cnt=175 pcode=0x1501:cnt=161 pcode=0x1516:cnt=147 pcode=0x150b:cnt=130, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:51:09.563609] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1021][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:51:09.563634] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:09.569564] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58403][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=27] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:51:09.572682] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:09.580927] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4015] clock out of order(cur_ts=1736326269580924, cur_ts_=1736326269560879, delta=20045) [2025-01-08 00:51:09.581130] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58374][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=63] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:51:09.596777] INFO [PALF] get_log (palf_handle_impl.cpp:4108) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4C-0-0] [lt=101] submit fetch log task success(ret=0, palf_id=1001, server="192.168.159.138:2882", fetch_type=0, prev_lsn={lsn:2161847}, start_lsn={lsn:2161960}, fetch_log_size=41943040, fetch_log_count=2048, accepted_mode_pid=1) [2025-01-08 00:51:09.598396] INFO [PALF] handle (fetch_log_engine.cpp:284) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=16] handle fetch_log_task(fetch_log_task={timestamp_us:1736326269596767, id:1001, server:"192.168.159.138:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:1, prev_lsn:{lsn:2161847}, start_lsn:{lsn:2161960}, log_size:41943040, log_count:2048, accepted_mode_pid:1}) [2025-01-08 00:51:09.598475] INFO [PALF] next (palf_iterator.h:183) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT ITERATOR INFO](io_ctx={user:"FETCHLOG", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:1, miss_cnt:0, cache_read_size:113, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:2161847}}}) [2025-01-08 00:51:09.598684] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:718) [50564][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=56] handle timeout(ret=0, *this={this:0x2ad8fb6c14d0, ref:4, trans_id:{txid:66699}, tenant_id:1001, is_exiting:false, trans_expired_time:1736326298733646, cluster_version:17180067072, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1736326269264110], ctx_create_time:1736326268733666}{self_ls_id:{id:1}, session_id:3221581057, associated_session_id:3221581057, part_trans_action:3, pending_write:0, 2pc_role:"UNKNOWN", ctx_tx_data_:{ctx_mgr_:0x2ad90c604030, tx_data_guard_:{tx_data:{tx_id:{txid:66699}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1736326268733666001, v:0}, start_scn:{val:1736326268733666001, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}}, read_only_:false}, role_state_:0, create_ctx_scn_:{val:18446744073709551615, v:3}, ctx_source:1, epoch_:470087889154523, replay_completeness_:1, upstream_state:"INIT", collected:[], rec_log_ts:{val:1736326268733666001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, last_request_ts:1736326268733666, block_frozen_memtable:null, max_2pc_commit_scn:{val:18446744073709551615, v:3}, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324362279616 max_table_version=1736324362279616 trans_version={val:1736326268733666001, v:0} commit_version={val:1736326268733666001, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:66699} ls_id=1 row_callback[alloc:1, free:0, unsubmit:0] redo[fill:2,sync_succ:0, sync_fail:0] main_list_len=2 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,2,2,0,2,0,0,0)|]}} -{busy_cbs_cnt:1, oldest_busy_cb:{ObTxBaseLogCb:{base_ts:{val:1736326268733666001, v:0}, log_ts:{val:1736326268733666001, v:0}, lsn:{lsn:43162003}, submit_ts:1736326269264287}, this:0x2ad8fb6c4ed8, is_inited_:true, trans_id:{txid:66699}, ls_id:{id:1}, ctx:0x2ad8fb6c14d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, final_log_cb:{ObTxBaseLogCb:{base_ts:{val:1736326268733666001, v:0}, log_ts:{val:1736326268733666001, v:0}, lsn:{lsn:43162003}, submit_ts:1736326269264287}, this:0x2ad8fb6c4ed8, is_inited_:true, trans_id:{txid:66699}, ls_id:{id:1}, ctx:0x2ad8fb6c14d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, -{info_log_submitted:1 state_log_submitting:1 state_log_submitted:1 }-, -{downstream_state:"INIT", upstream:{id:-1}, participants:[{id:1}], redo_log_no:0, scheduler:"192.168.159.137:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_appling_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:{branch:0, seq:3}, checksum:[0], checksum_scn:[{val:0, v:0}], need_checksum:true, data_complete:false, is_dup_tx:false, exec_epoch:0, commit_parts:[{ls_id:{id:1}, exec_epoch:470087889154523, transfer_epoch:-1}], }- }-}, tx_expired=false, commit_expired=false, delay=100000) [2025-01-08 00:51:09.598869] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:721) [50564][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=183] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1736326268733666001, v:0}, log_ts:{val:1736326268733666001, v:0}, lsn:{lsn:43162003}, submit_ts:1736326269264287}, this:0x2ad8fb6c4ed8, is_inited_:true, trans_id:{txid:66699}, ls_id:{id:1}, ctx:0x2ad8fb6c14d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1736326268733666001, v:0}, log_ts:{val:1736326268733666001, v:0}, lsn:{lsn:43162003}, submit_ts:1736326269264287}, this:0x2ad8fb6c4ed8, is_inited_:true, trans_id:{txid:66699}, ls_id:{id:1}, ctx:0x2ad8fb6c14d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}) [2025-01-08 00:51:09.598942] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:718) [50565][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=54] handle timeout(ret=0, *this={this:0x2ad8fb6fc5d0, ref:4, trans_id:{txid:66698}, tenant_id:1001, is_exiting:false, trans_expired_time:1736326297578008, cluster_version:17180067072, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1736326267820343], ctx_create_time:1736326267578022}{self_ls_id:{id:1}, session_id:3221581796, associated_session_id:3221581796, part_trans_action:3, pending_write:0, 2pc_role:"UNKNOWN", ctx_tx_data_:{ctx_mgr_:0x2ad90c604030, tx_data_guard_:{tx_data:{tx_id:{txid:66698}, ref_cnt:1, state:"ELR_COMMIT", commit_version:{val:1736326267578022001, v:0}, start_scn:{val:1736326267578022001, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}}, read_only_:false}, role_state_:0, create_ctx_scn_:{val:18446744073709551615, v:3}, ctx_source:1, epoch_:470086445559609, replay_completeness_:1, upstream_state:"INIT", collected:[], rec_log_ts:{val:1736326267578022001, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, last_request_ts:1736326267578022, block_frozen_memtable:null, max_2pc_commit_scn:{val:18446744073709551615, v:3}, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324362279616 max_table_version=1736324362279616 trans_version={val:1736326267578022001, v:0} commit_version={val:1736326267578022001, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:66698} ls_id=1 row_callback[alloc:1, free:0, unsubmit:0] redo[fill:2,sync_succ:0, sync_fail:0] main_list_len=2 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,2,2,0,2,0,0,0)|]}} -{busy_cbs_cnt:1, oldest_busy_cb:{ObTxBaseLogCb:{base_ts:{val:1736326267578022001, v:0}, log_ts:{val:1736326267578022001, v:0}, lsn:{lsn:43161349}, submit_ts:1736326267820379}, this:0x2ad8fb6fffd8, is_inited_:true, trans_id:{txid:66698}, ls_id:{id:1}, ctx:0x2ad8fb6fc5d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, final_log_cb:{ObTxBaseLogCb:{base_ts:{val:1736326267578022001, v:0}, log_ts:{val:1736326267578022001, v:0}, lsn:{lsn:43161349}, submit_ts:1736326267820379}, this:0x2ad8fb6fffd8, is_inited_:true, trans_id:{txid:66698}, ls_id:{id:1}, ctx:0x2ad8fb6fc5d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, -{info_log_submitted:1 state_log_submitting:1 state_log_submitted:1 }-, -{downstream_state:"INIT", upstream:{id:-1}, participants:[{id:1}], redo_log_no:0, scheduler:"192.168.159.137:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_appling_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:{branch:0, seq:3}, checksum:[0], checksum_scn:[{val:0, v:0}], need_checksum:true, data_complete:false, is_dup_tx:false, exec_epoch:0, commit_parts:[{ls_id:{id:1}, exec_epoch:470086445559609, transfer_epoch:-1}], }- }-}, tx_expired=false, commit_expired=false, delay=100000) [2025-01-08 00:51:09.599053] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:721) [50565][T1001_TransTime][T1001][Y0-0000000000000000-0-0] [lt=109] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1736326267578022001, v:0}, log_ts:{val:1736326267578022001, v:0}, lsn:{lsn:43161349}, submit_ts:1736326267820379}, this:0x2ad8fb6fffd8, is_inited_:true, trans_id:{txid:66698}, ls_id:{id:1}, ctx:0x2ad8fb6fc5d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1736326267578022001, v:0}, log_ts:{val:1736326267578022001, v:0}, lsn:{lsn:43161349}, submit_ts:1736326267820379}, this:0x2ad8fb6fffd8, is_inited_:true, trans_id:{txid:66698}, ls_id:{id:1}, ctx:0x2ad8fb6fc5d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}) [2025-01-08 00:51:09.599306] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:718) [46310][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=190] handle timeout(ret=0, *this={this:0x2ad89c082950, ref:5, trans_id:{txid:27912}, tenant_id:1, is_exiting:false, trans_expired_time:1736326298538661, cluster_version:17180067072, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1736326268757661], ctx_create_time:1736326268538680}{self_ls_id:{id:1}, session_id:3221581848, associated_session_id:3221581848, part_trans_action:3, pending_write:0, 2pc_role:"UNKNOWN", ctx_tx_data_:{ctx_mgr_:0x2ad8afe04030, tx_data_guard_:{tx_data:{tx_id:{txid:27912}, ref_cnt:1, state:"RUNNING", commit_version:{val:1736326268538680000, v:0}, start_scn:{val:1736326268538680000, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}}, read_only_:false}, role_state_:0, create_ctx_scn_:{val:18446744073709551615, v:3}, ctx_source:1, epoch_:470087382515190, replay_completeness_:1, upstream_state:"INIT", collected:[], rec_log_ts:{val:1736326268538680000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, last_request_ts:1736326268538680, block_frozen_memtable:null, max_2pc_commit_scn:{val:18446744073709551615, v:3}, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322553629032 max_table_version=1736322553629032 trans_version={val:1736326268538680000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27912} ls_id=1 row_callback[alloc:1, free:0, unsubmit:0] redo[fill:2,sync_succ:0, sync_fail:0] main_list_len=2 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,2,2,0,2,0,0,0)|]}} -{busy_cbs_cnt:1, oldest_busy_cb:{ObTxBaseLogCb:{base_ts:{val:1736326268538680000, v:0}, log_ts:{val:1736326268538680000, v:0}, lsn:{lsn:84248733}, submit_ts:1736326268757758}, this:0x2ad89c086358, is_inited_:true, trans_id:{txid:27912}, ls_id:{id:1}, ctx:0x2ad89c082950, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, final_log_cb:{ObTxBaseLogCb:{base_ts:{val:1736326268538680000, v:0}, log_ts:{val:1736326268538680000, v:0}, lsn:{lsn:84248733}, submit_ts:1736326268757758}, this:0x2ad89c086358, is_inited_:true, trans_id:{txid:27912}, ls_id:{id:1}, ctx:0x2ad89c082950, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, -{info_log_submitted:1 state_log_submitting:1 state_log_submitted:1 }-, -{downstream_state:"INIT", upstream:{id:-1}, participants:[{id:1}], redo_log_no:0, scheduler:"192.168.159.137:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_appling_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:{branch:0, seq:3}, checksum:[0], checksum_scn:[{val:0, v:0}], need_checksum:true, data_complete:false, is_dup_tx:false, exec_epoch:0, commit_parts:[{ls_id:{id:1}, exec_epoch:470087382515190, transfer_epoch:-1}], }- }-}, tx_expired=false, commit_expired=false, delay=100000) [2025-01-08 00:51:09.599443] INFO [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:721) [46310][T1_TransTimeWhe][T1][Y0-0000000000000000-0-0] [lt=136] trx is waiting log_cb(busy_cbs_.get_size()=1, busy_cbs_.get_first()={ObTxBaseLogCb:{base_ts:{val:1736326268538680000, v:0}, log_ts:{val:1736326268538680000, v:0}, lsn:{lsn:84248733}, submit_ts:1736326268757758}, this:0x2ad89c086358, is_inited_:true, trans_id:{txid:27912}, ls_id:{id:1}, ctx:0x2ad89c082950, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, busy_cbs_.get_last()={ObTxBaseLogCb:{base_ts:{val:1736326268538680000, v:0}, log_ts:{val:1736326268538680000, v:0}, lsn:{lsn:84248733}, submit_ts:1736326268757758}, this:0x2ad89c086358, is_inited_:true, trans_id:{txid:27912}, ls_id:{id:1}, ctx:0x2ad89c082950, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}) [2025-01-08 00:51:09.599786] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:09.600059] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326265444937, last_first_register_time_us=1736324454612308, register_time_us=1736326265444937, curr_time_us=1736326269600056) [2025-01-08 00:51:09.600148] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=41] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326269600101, last_submit_register_req_time_us=1736326269600101, register_parent_reason=1) [2025-01-08 00:51:09.600198] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=48] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [46142]OB_MALLOC COST TOO MUCH TIME, cost_time=285771, tenant_id=1002, label=CoStack, ctx_id=8, prio=1, size=515136, owner: ObMalloc, click_count: 1, time dist:[ALLOC_CHUNK_END=285768], seq:[0] [2025-01-08 00:51:09.605473] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326269605437], gts:1736326269604896000, latest_srr:[mts=1736326269605437]}) [2025-01-08 00:51:09.605535] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=41] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326269605517], gts:1736326269604896000, latest_srr:[mts=1736326269605517]}) [2025-01-08 00:51:09.605772] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326269605759], gts:1736326269604896000, latest_srr:[mts=1736326269605759]}) [2025-01-08 00:51:09.618537] INFO [PALF] next (palf_iterator.h:183) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT ITERATOR INFO](io_ctx={user:"FETCHLOG", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:1, miss_cnt:0, cache_read_size:122, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:2161960}}}) [2025-01-08 00:51:09.618618] WDIAG [PALF] batch_fetch_log_each_round_ (palf_handle_impl.cpp:3882) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=75][errcode=-4008] iterator next failed(ret=-4008, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:0x2ad924ecf000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2161960}, end_lsn:{lsn:2162082}, read_buf:{buf_len_:73728, buf_:0x2ad924ecf000}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2162082}, readable_log_tail:{lsn:2162082}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64942686, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326267132611004, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:2, group_size:66, proposal_id:1, committed_lsn:{lsn:2161960}, max_scn:{val:1736326267132611004, v:0}, accumulated_checksum:3669121729, log_id:17763, flag:59712}}, init_mode_version:0, accumulate_checksum:3669121729, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326267132611004, v:0}}, io_ctx:{user:"FETCHLOG", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:1, miss_cnt:0, cache_read_size:122, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:2161960}}}}, replayable_point={val:1736326264665540004, v:0}, prev_lsn={lsn:2161960}) [2025-01-08 00:51:09.618738] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3819) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=69] fetch_log_from_storage_ finished(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, prev_lsn={lsn:2161847}, iterator={iterator_impl:{buf_:0x2ad924ecf000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2161960}, end_lsn:{lsn:2162082}, read_buf:{buf_len_:73728, buf_:0x2ad924ecf000}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2162082}, readable_log_tail:{lsn:2162082}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64942686, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326267132611004, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:2, group_size:66, proposal_id:1, committed_lsn:{lsn:2161960}, max_scn:{val:1736326267132611004, v:0}, accumulated_checksum:3669121729, log_id:17763, flag:59712}}, init_mode_version:0, accumulate_checksum:3669121729, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326267132611004, v:0}}, io_ctx:{user:"FETCHLOG", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:1, miss_cnt:0, cache_read_size:122, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:2161960}}}}, fetch_log_count=2048, has_fetched_log_count=1) [2025-01-08 00:51:09.618849] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=80] submit_change_mode_meta_req success(ret=0, palf_id=1001, server="192.168.159.138:2882", mode_meta={version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, accepted_mode_pid=1) [2025-01-08 00:51:09.619127] INFO [PALF] fetch_log_from_storage (palf_handle_impl.cpp:3601) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=277] [PALF STAT FETCH LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=83886080) [2025-01-08 00:51:09.619142] INFO [LIB] stat (utility.h:1210) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FETCH LOG TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1626, this=0x2ad8f0ff8600, extra_info="") [2025-01-08 00:51:09.619153] INFO [LIB] stat (utility.h:1210) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FETCH LOG EXECUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=20748, this=0x2ad8f0ff8830, extra_info="") [2025-01-08 00:51:09.619159] INFO [PALF] handle (fetch_log_engine.cpp:321) [50431][T1002_FetchLog0][T1002][Y0-0000000000000000-0-0] [lt=6] handle fetch log task(ret=0, palf_id=1001, handle_cost_time_us=20748, fetch_log_task={timestamp_us:1736326269596767, id:1001, server:"192.168.159.138:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:1, prev_lsn:{lsn:2161847}, start_lsn:{lsn:2161960}, log_size:41943040, log_count:2048, accepted_mode_pid:1}, fetch_stat={total_size:122, group_log_cnt:1, read_cost:0, get_cost:0, send_cost:44}) [2025-01-08 00:51:09.647885] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58426][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:09.648056] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29E-0-0] [lt=53] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:09.648142] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29E-0-0] [lt=63] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326267147675004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326267147675004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:09.649691] INFO [SERVER.OMT] recv_group_request (ob_tenant.cpp:1413) [46142][pnio1][T0][YB42C0A89F8A-00062B2D1A79C29E-0-0] [lt=54] worker thread created(id()=1002, group->group_id_=18) [2025-01-08 00:51:09.649777] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F8A-00062B2D16C9BEC7-0-0] [lt=69][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1308, fly_ts=256971, send_timestamp=1736326269392803, peer="192.168.159.138:42000", sz=1728) [2025-01-08 00:51:09.665822] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=0] ====== checkpoint timer task ====== [2025-01-08 00:51:09.666593] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=19] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:2, active_tx_count:2, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:09.666635] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=41] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1736326267578022001, v:0}) [2025-01-08 00:51:09.666821] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279284252000, latest_log_ts:{val:1736326262663505001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:09.666906] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=81] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:09.666936] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:09.666951] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326262663505001, v:0}) [2025-01-08 00:51:09.666959] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:09.666966] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:09.666979] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:09.667648] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:09.667676] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:09.667699] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:09.667722] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:09.667732] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:09.669557] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=67] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:09.674059] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=62] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:09.674093] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:09.678766] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84980-0-0] [lt=32] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:09.678913] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84980-0-0] [lt=97] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:09.685653] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30] start to print blacklist info [2025-01-08 00:51:09.686361] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=176] ls blacklist refresh finish(cost_time=432869) [2025-01-08 00:51:09.688195] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:09.690582] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58408][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=22] worker thread exit(tenant_->id()=1002, workers_.get_size()=6) [2025-01-08 00:51:09.699069] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46382][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB90-0-0] [lt=25][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774612900, cache_obj->added_lc()=false, cache_obj->get_object_id()=13601, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.701664] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:09.702168] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708514F-0-0] [lt=17] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:09.702361] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085151-0-0] [lt=21] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:09.704815] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [58409][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1736326269697243000, start_id=1736326269697243000, end_id=1736326269697243001) [2025-01-08 00:51:09.705110] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46308][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=72] expand thread count(this=0x2ad8881fc3b0, max_thread_cnt=128, cur_thread_count=5, inc_cnt=1, queue_size=5) [2025-01-08 00:51:09.705780] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=40] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:32K/8:384591690 disk:32K/8:384591690 t1m2:4096/1:1130021190 t1m2g0:4096/1:1130021190 t1002m2:16K/4:250662809 t1002m2g0:16K/4:250662809 t1001m2:12K/3:314604127 t1001m2g0:12K/3:314604127 |q0: 0 1 0 |q1: 0 0 0 [2025-01-08 00:51:09.705751] WDIAG [STORAGE.TRANS] commit (ob_trans_part_ctx.cpp:844) [46381][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1E29B9E4-0-0] [lt=452][errcode=-4023] tx is 2pc logging(ret=-4023, ret="OB_EAGAIN", this={this:0x2ad89c086cd0, ref:3, trans_id:{txid:55693}, tenant_id:1, is_exiting:false, trans_expired_time:1736412669513856, cluster_version:17180067072, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1736326269558565], ctx_create_time:1736326269545291}{self_ls_id:{id:1}, session_id:3221814394, associated_session_id:0, part_trans_action:3, pending_write:0, 2pc_role:"UNKNOWN", ctx_tx_data_:{ctx_mgr_:0x2ad8afe04030, tx_data_guard_:{tx_data:{tx_id:{txid:55693}, ref_cnt:1, state:"RUNNING", commit_version:{val:1736326269558308000, v:0}, start_scn:{val:1736326269558308000, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}}, read_only_:false}, role_state_:0, create_ctx_scn_:{val:18446744073709551615, v:3}, ctx_source:1, epoch_:470088173998968, replay_completeness_:1, upstream_state:"INIT", collected:[], rec_log_ts:{val:1736326269558308000, v:0}, prev_rec_log_ts:{val:18446744073709551615, v:3}, lastest_snapshot:{val:18446744073709551615, v:3}, last_request_ts:1736326269558308, block_frozen_memtable:null, max_2pc_commit_scn:{val:18446744073709551615, v:3}, mt_ctx_:{ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322603555848 max_table_version=1736322603555848 trans_version={val:1736326269558308000, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55693} ls_id=1 row_callback[alloc:1, free:0, unsubmit:0] redo[fill:2,sync_succ:0, sync_fail:0] main_list_len=2 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,2,2,0,2,0,0,0)|]}} -{busy_cbs_cnt:1, oldest_busy_cb:{ObTxBaseLogCb:{base_ts:{val:1736326269558308000, v:0}, log_ts:{val:1736326269558308000, v:0}, lsn:{lsn:84249752}, submit_ts:1736326269558697}, this:0x2ad89c08a6d8, is_inited_:true, trans_id:{txid:55693}, ls_id:{id:1}, ctx:0x2ad89c086cd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, final_log_cb:{ObTxBaseLogCb:{base_ts:{val:1736326269558308000, v:0}, log_ts:{val:1736326269558308000, v:0}, lsn:{lsn:84249752}, submit_ts:1736326269558697}, this:0x2ad89c08a6d8, is_inited_:true, trans_id:{txid:55693}, ls_id:{id:1}, ctx:0x2ad89c086cd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, -{info_log_submitted:1 state_log_submitting:1 state_log_submitted:1 }-, -{downstream_state:"INIT", upstream:{id:-1}, participants:[{id:1}], redo_log_no:0, scheduler:"192.168.159.138:2882", prepare_version:{val:18446744073709551615, v:3}, trans_type:0, next_log_entry_no:1, max_applied_log_ts:{val:18446744073709551615, v:3}, max_appling_log_ts:{val:18446744073709551615, v:3}, max_applying_part_log_no:9223372036854775807, max_submitted_seq_no:{branch:0, seq:115867}, checksum:[0], checksum_scn:[{val:0, v:0}], need_checksum:true, data_complete:false, is_dup_tx:false, exec_epoch:0, commit_parts:[{ls_id:{id:1}, exec_epoch:470088173998968, transfer_epoch:-1}], }- }-}) [2025-01-08 00:51:09.705891] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:2048) [46381][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1E29B9E4-0-0] [lt=138][errcode=-4023] commit fail(ret=-4023, coord={id:1}, tx_id={txid:55693}) [2025-01-08 00:51:09.705905] WDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1977) [46381][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1E29B9E4-0-0] [lt=12][errcode=-4023] handle trans commit request failed(ret=-4023, msg={txMsg:{type:20, cluster_version:17180067072, tenant_id:1, tx_id:{txid:55693}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"192.168.159.138:2882", epoch:470088173998968, transfer_epoch:-1, request_id:4, timestamp:1736326269784273, cluster_id:1736322126}, expire_ts:1736326279648603, commit_start_scn:{val:1736326269508251000, v:0}, parts:[{id:1}], app_trace_info:"", commit_parts:[{ls_id:{id:1}, exec_epoch:470088173998968, transfer_epoch:-1}]}) [2025-01-08 00:51:09.705946] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.cpp:149) [46381][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1E29B9E4-0-0] [lt=40][errcode=-4023] handle txn message fail(ret=-4023, ret="OB_EAGAIN", msg={txMsg:{type:20, cluster_version:17180067072, tenant_id:1, tx_id:{txid:55693}, receiver:{id:1}, sender:{id:9223372036854775807}, sender_addr:"192.168.159.138:2882", epoch:470088173998968, transfer_epoch:-1, request_id:4, timestamp:1736326269784273, cluster_id:1736322126}, expire_ts:1736326279648603, commit_start_scn:{val:1736326269508251000, v:0}, parts:[{id:1}], app_trace_info:"", commit_parts:[{ls_id:{id:1}, exec_epoch:470088173998968, transfer_epoch:-1}]}) [2025-01-08 00:51:09.706497] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:09.706829] INFO [LIB] do_thread_recycle (threads.cpp:163) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=34] recycle one thread(this=0x2ad8f90b0e70, total=7, remain=6) [2025-01-08 00:51:09.706853] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=19] expand thread count(this=0x2ad8f90b0e70, max_thread_cnt=128, cur_thread_count=6, inc_cnt=1, queue_size=6) [2025-01-08 00:51:09.707308] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=101] Slow write(duration=426814, write_len=4096) [2025-01-08 00:51:09.707326] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=15] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8625, offset=2447495}, end_cursor:ObLogCursor{file_id=1, log_id=8626, offset=2447800}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947b4050, len:1913, log_data_len:305, seq:8625, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:09.707308] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=65] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6718582784, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:09.707495] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18670443, this={dio_aligned_buf:{buf_write_offset:997, buf_padding_size:3099, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:32305, truncate_used_ts:6686}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=18669568, buf_len=122, total_write_size=12789, total_write_size_after_dio=348160, ob_pwrite_used_ts=2244372, count=82, each_pwrite_cost=27370) [2025-01-08 00:51:09.707527] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=231970, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:09.707539] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18666347}], scn_array=[cnt:1, {val:1736326267890517004, v:0}], curr_size=122, accum_size=479, time_cost=1271642) [2025-01-08 00:51:09.707558] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=479) [2025-01-08 00:51:09.707575] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=799, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:09.707682] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.137:2882", update_func={old_end_lsn:{lsn:18666347}, new_end_lsn:{lsn:18666469}, old_advance_time_us:1736326268360289, new_ack_time_us:1736326269707678, advance delay(us):1347389}) [2025-01-08 00:51:09.707752] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=31, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:09.707774] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=3, avg_log_batch_cnt=1, total_group_log_size=845, avg_group_log_size=281) [2025-01-08 00:51:09.707813] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:09.707825] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG LIFE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1273305, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:09.707784] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [58391][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=0] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5217164325, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:09.707832] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=6] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=1, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=299, avg_log_submit_to_flush_time=1272811, avg_log_submit_to_slide_time=1273006) [2025-01-08 00:51:09.707843] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=19397, log_task={header:{begin_lsn:{lsn:18666347}, end_lsn:{lsn:18666469}, log_id:19397, min_scn:{val:1736326267890517004, v:0}, max_scn:{val:1736326267890517004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18666347}, data_checksum:667309872, accum_checksum:2328259632, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326268434485, freeze_ts:1736326268434494, submit_ts:1736326268434784, flushed_ts:1736326269707595, gen_to_freeze cost time:9, gen_to_submit cost time:299, submit_to_flush cost time:1272811}, fs_cb_begin_ts=1736326269707790, log_life_time=1273305) [2025-01-08 00:51:09.707915] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=319, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:09.707956] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=122, hit_cnt=1, read_cnt=1, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:09.708395] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=18] [APPLY STAT CB APPEND COST TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=190, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:09.708411] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=16] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=136180, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:09.708419] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:09.708426] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB EXECUTE TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=-1843, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:09.708509] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=81] [APPLY STAT CB TOTAL TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=134526, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:09.708518] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:948) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] cb cost too much time(lsn={lsn:18666403}, scn={val:1736326267890517004, v:0}, idx=12, total_cost_time=1273412, append_cost_time=494, cb_wait_thread_time=1273573, cb_wait_commit_time=0, cb_cost_time=-655, append_start_time=1736326268434324, append_finish_time=1736326268434818, cb_first_handle_time=1736326269708391, cb_finish_time=1736326269707736) [2025-01-08 00:51:09.708865] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=198616, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:09.708949] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2182596}], scn_array=[cnt:1, {val:1736326267890517000, v:0}], curr_size=113, accum_size=113, time_cost=198616) [2025-01-08 00:51:09.708990] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=113) [2025-01-08 00:51:09.709019] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=250, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:09.709030] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=3, this=0x2ad8f0fff7a8, extra_info="log_io_worker_num:4, cb_thread_pool_tg_id:344") [2025-01-08 00:51:09.709297] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17932, log_task={header:{begin_lsn:{lsn:2182596}, end_lsn:{lsn:2182709}, log_id:17932, min_scn:{val:1736326267890517000, v:0}, max_scn:{val:1736326267890517000, v:0}, data_len:57, proposal_id:1, prev_lsn:{lsn:2182474}, prev_proposal_id:1, committed_end_lsn:{lsn:2182596}, data_checksum:1133849145, accum_checksum:3129141197, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326269505758, freeze_ts:1736326269505758, submit_ts:1736326269509994, flushed_ts:1736326269709101, gen_to_freeze cost time:0, gen_to_submit cost time:4236, submit_to_flush cost time:199107}, fs_cb_begin_ts=1736326269709275, log_life_time=203517) [2025-01-08 00:51:09.709329] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=228, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:09.709942] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=209] succ to submit log task to replay service(to_submit_lsn={lsn:2182652}, to_submit_scn={val:1736326267890517000, v:0}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326269709279, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2182709}, next_to_submit_scn_:{val:1736326267890517001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:113, curr_read_buf_start_pos:0, curr_read_buf_end_pos:113, log_storage_:{start_lsn:{lsn:2182596}, end_lsn:{lsn:2182709}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2182709}, readable_log_tail:{lsn:2182709}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64922059, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326267890517000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:25, scn_:{val:1736326267890517000, v:0}, data_checksum:1198461532, flag:2943}}, init_mode_version:0, accumulate_checksum:3129141197, curr_entry_is_padding:0, padding_entry_size:57, padding_entry_scn:{val:1736326267890517000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17843, miss_cnt:0, cache_read_size:2182709, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:09.710631] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2162082}], scn_array=[cnt:1, {val:1736326268733666000, v:0}], curr_size=122, accum_size=122, time_cost=321150) [2025-01-08 00:51:09.710921] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=17764, log_task={header:{begin_lsn:{lsn:2162082}, end_lsn:{lsn:2162204}, log_id:17764, min_scn:{val:1736326268733666000, v:0}, max_scn:{val:1736326268733666000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2162082}, data_checksum:2222604892, accum_checksum:3613587091, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326269389305, freeze_ts:1736326269389315, submit_ts:1736326269389407, flushed_ts:1736326269710889, gen_to_freeze cost time:10, gen_to_submit cost time:102, submit_to_flush cost time:321482}, fs_cb_begin_ts=1736326269710898, log_life_time=321593) [2025-01-08 00:51:09.711351] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=947149, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:09.711437] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=84][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43161293}], scn_array=[cnt:1, {val:1736326267578022001, v:0}], curr_size=654, accum_size=654, time_cost=947149) [2025-01-08 00:51:09.711463] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=654) [2025-01-08 00:51:09.711467] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F4-0-0] [lt=408] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:09.711483] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=30061, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:09.711506] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F4-0-0] [lt=23] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326267890517000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326267890517000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:09.711511] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=654) [2025-01-08 00:51:09.711587] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=1, avg_log_batch_cnt=1, total_group_log_size=589, avg_group_log_size=589) [2025-01-08 00:51:09.711645] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=1, avg_log_gen_to_freeze_time=913648, avg_log_gen_to_submit_time=913751, avg_log_submit_to_flush_time=977369, avg_log_submit_to_slide_time=977501) [2025-01-08 00:51:09.711658] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=32535, log_task={header:{begin_lsn:{lsn:43161293}, end_lsn:{lsn:43161947}, log_id:32535, min_scn:{val:1736326267578022001, v:0}, max_scn:{val:1736326267834686000, v:0}, data_len:598, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43161293}, data_checksum:589953219, accum_checksum:363697338, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326267820384, freeze_ts:1736326268734032, submit_ts:1736326268734135, flushed_ts:1736326269711504, gen_to_freeze cost time:913648, gen_to_submit cost time:913751, submit_to_flush cost time:977369}, fs_cb_begin_ts=1736326269711636, log_life_time=1891252) [2025-01-08 00:51:09.711707] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=654, hit_cnt=1, read_cnt=1, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:09.711741] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=66][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326267578022001, v:0}, log_ts:{val:1736326267578022001, v:0}, lsn:{lsn:43161349}, submit_ts:1736326267820379}, this:0x2ad8fb6fffd8, is_inited_:true, trans_id:{txid:66698}, ls_id:{id:1}, ctx:0x2ad8fb6fc5d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=1891359, ctx_lock_wait_time=1) [2025-01-08 00:51:09.711799] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=32] long trans statistics(cur_stat_count=4, stat_interval=60000000, avg (count/cost)=0, this=0x5572d8b02280, extra_info="") [2025-01-08 00:51:09.712090] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326267354573000, v:0}, new_max_version={val:1736326267354573000, v:0}, last_min_version={val:1736326267282501000, v:0}, last_max_version={val:1736326267282501000, v:0}, total_time_us=1893444, delta_us=2357515) [2025-01-08 00:51:09.712115] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326267354573000, v:0}, min_version={val:1736326267354573000, v:0}, max_version={val:1736326267354573000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:09.712832] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326267470393001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=2242431, in_cluster_service=true, cluster_version={val:1736326267354573000, v:0}, min_cluster_version={val:1736326267354573000, v:0}, max_cluster_version={val:1736326267354573000, v:0}, get_cluster_version_err=0, cluster_version_delta=2358251, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18680, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=615349607, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:09.712867] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:09.713549] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1138283, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:09.713573] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84248555}], scn_array=[cnt:1, {val:1736326267578022003, v:0}], curr_size=122, accum_size=122, time_cost=1138283) [2025-01-08 00:51:09.713590] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=122) [2025-01-08 00:51:09.713603] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1731, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:09.713628] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.137:2882", update_func={old_end_lsn:{lsn:84248555}, new_end_lsn:{lsn:84248677}, old_advance_time_us:1736326268573384, new_ack_time_us:1736326269713626, advance delay(us):1140242}) [2025-01-08 00:51:09.713740] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=83, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:09.713761] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=1, avg_log_batch_cnt=3, total_group_log_size=1583, avg_group_log_size=1583) [2025-01-08 00:51:09.713782] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:09.713838] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=54] [PALF STAT LOG LIFE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1635066, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:09.713850] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=1, avg_log_gen_to_freeze_time=494728, avg_log_gen_to_submit_time=494822, avg_log_submit_to_flush_time=1140092, avg_log_submit_to_slide_time=1140244) [2025-01-08 00:51:09.713859] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=44561, log_task={header:{begin_lsn:{lsn:84248555}, end_lsn:{lsn:84248677}, log_id:44561, min_scn:{val:1736326267578022003, v:0}, max_scn:{val:1736326267578022003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84248555}, data_checksum:667309872, accum_checksum:704720349, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326268078709, freeze_ts:1736326268573437, submit_ts:1736326268573531, flushed_ts:1736326269713623, gen_to_freeze cost time:494728, gen_to_submit cost time:494822, submit_to_flush cost time:1140092}, fs_cb_begin_ts=1736326269713775, log_life_time=1635066) [2025-01-08 00:51:09.713892] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=269, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:09.713926] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=122, hit_cnt=1, read_cnt=1, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:09.718536] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4B-0-0] [lt=19] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:09.718594] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4B-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326267890517004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326267890517004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:09.724364] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=27][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326268538680000, v:0}, log_ts:{val:1736326268538680000, v:0}, lsn:{lsn:84248733}, submit_ts:1736326268757758}, this:0x2ad89c086358, is_inited_:true, trans_id:{txid:27912}, ls_id:{id:1}, ctx:0x2ad89c082950, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=966602, ctx_lock_wait_time=1) [2025-01-08 00:51:09.724461] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=52][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326269558308000, v:0}, log_ts:{val:1736326269558308000, v:0}, lsn:{lsn:84249752}, submit_ts:1736326269558697}, this:0x2ad89c08a6d8, is_inited_:true, trans_id:{txid:55693}, ls_id:{id:1}, ctx:0x2ad89c086cd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=165762, ctx_lock_wait_time=0) [2025-01-08 00:51:09.724616] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=25] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:09.724640] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=13][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326269366594000, v:0}, log_ts:{val:1736326269366594000, v:0}, lsn:{lsn:84249222}, submit_ts:1736326269410557}, this:0x2ad89c0a7f58, is_inited_:true, trans_id:{txid:27913}, ls_id:{id:1}, ctx:0x2ad89c0a4550, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=314081, ctx_lock_wait_time=1) [2025-01-08 00:51:09.729246] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324465595928000) [2025-01-08 00:51:09.737799] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46386][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDD7-0-0] [lt=47][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774651632, cache_obj->added_lc()=false, cache_obj->get_object_id()=13602, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.740290] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2C63-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=15, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:09.740317] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2C63-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=1183608, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:09.740402] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326268733666001, v:0}, log_ts:{val:1736326268733666001, v:0}, lsn:{lsn:43162003}, submit_ts:1736326269264287}, this:0x2ad8fb6c4ed8, is_inited_:true, trans_id:{txid:66699}, ls_id:{id:1}, ctx:0x2ad8fb6c14d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=476111, ctx_lock_wait_time=1) [2025-01-08 00:51:09.741084] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=30] submit reserved snapshot log success(ls_id={id:1001}, new_snapshot_version=1736324465595928000) [2025-01-08 00:51:09.741152] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] update reserved snapshot finished(cost_ts=251349, reserved_snapshot=1736324465595928000) [2025-01-08 00:51:09.754388] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C29E-0-0] [lt=79] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=2, total_release_part_ctx_count=73) [2025-01-08 00:51:09.759734] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:491) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E4B-0-0] [lt=22] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=0, MAX_GAP=900000000000, new_sync_scn={val:1736326267132611001, v:0}, new_replayable_scn={val:1736326267132611001, v:0}, new_readable_scn={val:1736326267132611001, v:0}, old_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326266130647000, replayable_scn:1736326266130647000, readable_scn:1736326265815330002, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}) [2025-01-08 00:51:09.760275] INFO [LIB] stat (utility.h:1210) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E4B-0-0] [lt=33] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=68, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:09.760323] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E4B-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=589) [2025-01-08 00:51:09.760651] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1736326269760648, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004519, add_bytes: 0 [2025-01-08 00:51:09.767206] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=73, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:09.767466] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=158] [APPLY STAT CB APPEND COST TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=123, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:09.767488] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=21] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=63629, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:09.767497] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:09.767504] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=-11937, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:09.767511] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=51816, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:09.769654] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D038-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:09.773132] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=17] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8626, offset=2447800}, end_cursor:ObLogCursor{file_id=1, log_id=8627, offset=2448097}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947bc050, len:1608, log_data_len:297, seq:8626, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:09.773551] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4B-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:09.773596] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4B-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326268733666001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326268733666001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:09.779942] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:09.780812] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:09.781659] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=471] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:09.787280] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=74] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:6, threads_idle_time:14991979587, tenant_id:1}, cur_thread_count=6, cnt=1, new_thread_count=7) [2025-01-08 00:51:09.787489] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:09.787519] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=24] ====== check clog disk timer task ====== [2025-01-08 00:51:09.787533] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=8] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:09.797260] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46382][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDDB-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774711091, cache_obj->added_lc()=false, cache_obj->get_object_id()=13603, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.800033] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:09.801363] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [58429][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:09.803321] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:09.803663] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [58429][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326269797549000, start_id=1736326269797549000, end_id=1736326269797549001) [2025-01-08 00:51:09.804338] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=76] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:7, threads_idle_time:7753154738, tenant_id:1001}, cur_thread_count=7, cnt=1, new_thread_count=8) [2025-01-08 00:51:09.804566] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=38] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=16106127, size_used=0, mem_used=16637952) [2025-01-08 00:51:09.806588] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=2, elapse_time=2021, size_used=0, mem_used=0) [2025-01-08 00:51:09.826486] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4015] clock out of order(cur_ts=1736326269826481, cur_ts_=1736326269797549, delta=28932) [2025-01-08 00:51:09.828745] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2162316}], scn_array=[cnt:1, {val:1736326269697243000, v:0}], curr_size=122, accum_size=356, time_cost=25005) [2025-01-08 00:51:09.830655] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8627, offset=2448097}, end_cursor:ObLogCursor{file_id=1, log_id=8628, offset=2448394}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947b8050, len:1311, log_data_len:297, seq:8627, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:09.832711] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=112] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8628, offset=2448394}, end_cursor:ObLogCursor{file_id=1, log_id=8629, offset=2448691}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947c0050, len:1014, log_data_len:297, seq:8628, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:09.836836] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2183075}], scn_array=[cnt:1, {val:1736326269768584001, v:0}], curr_size=122, accum_size=488, time_cost=10452) [2025-01-08 00:51:09.849088] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C626-0-0] [lt=31][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193774762919, cache_obj->added_lc()=false, cache_obj->get_object_id()=13605, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:09.851146] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=21] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:09.852805] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326269851345000, start_id=1736326269851345000, end_id=1736326269851345001) [2025-01-08 00:51:09.853827] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [58391][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=51] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5217550762, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:09.864599] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856DF-0-0] [lt=32] MediumCheckTask(cost_ts=8) [2025-01-08 00:51:09.868721] INFO [RPC] process (ob_batch_processor.cpp:106) [58384][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D22084595-0-0] [lt=45] batch rpc statistics(clog_batch_nodelay_cnt=0, clog_batch_cnt=1, trx_batch_cnt=0, sql_batch_cnt=0) [2025-01-08 00:51:09.870450] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46493][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB849-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326269868685000, start_id=1736326269868685000, end_id=1736326269868685001) [2025-01-08 00:51:09.873438] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2195239}], scn_array=[cnt:1, {val:1736326267890517002, v:0}], curr_size=122, accum_size=122, time_cost=550644) [2025-01-08 00:51:09.873566] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f0fff068, extra_info="log_io_worker_num:4, cb_thread_pool_tg_id:344") [2025-01-08 00:51:09.873623] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43166306}], scn_array=[cnt:1, {val:1736326269851345000, v:0}], curr_size=589, accum_size=4948, time_cost=20197) [2025-01-08 00:51:09.874100] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18036, log_task={header:{begin_lsn:{lsn:2195239}, end_lsn:{lsn:2195361}, log_id:18036, min_scn:{val:1736326267890517002, v:0}, max_scn:{val:1736326267890517002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2195117}, prev_proposal_id:1, committed_end_lsn:{lsn:2195239}, data_checksum:1715074492, accum_checksum:3482445571, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326269318180, freeze_ts:1736326269318180, submit_ts:1736326269318293, flushed_ts:1736326269874053, gen_to_freeze cost time:0, gen_to_submit cost time:113, submit_to_flush cost time:555760}, fs_cb_begin_ts=1736326269874086, log_life_time=555906) [2025-01-08 00:51:09.875305] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C29E-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1736326269871265000, start_id=1736326269871265000, end_id=1736326269871265001) [2025-01-08 00:51:09.886171] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84250846}], scn_array=[cnt:1, {val:1736326269868685000, v:0}], curr_size=1204, accum_size=3373, time_cost=15205) [2025-01-08 00:51:09.889643] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43166895}], scn_array=[cnt:1, {val:1736326269871265000, v:0}], curr_size=750, accum_size=5698, time_cost=13549) [2025-01-08 00:51:09.889761] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:6, {lsn:2195361}, {lsn:2195483}, {lsn:2195605}, {lsn:2195727}, {lsn:2195849}, {lsn:2195971}], scn_array=[cnt:6, {val:1736326267890517003, v:0}, {val:1736326267890517004, v:0}, {val:1736326267890517005, v:0}, {val:1736326269313726001, v:0}, {val:1736326269649674001, v:0}, {val:1736326269649674002, v:0}], curr_size=732, accum_size=854, time_cost=16169) [2025-01-08 00:51:09.897164] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.897267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.897295] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.897318] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:09.898021] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=118] ====== checkpoint timer task ====== [2025-01-08 00:51:09.898875] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=21] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:09.898918] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=43] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:09.899016] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279254996000, latest_log_ts:{val:1736326262937841001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:09.899099] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=78] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279254996000, latest_log_ts:{val:1736326262937841001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:09.899156] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=51] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:09.899170] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326262937841001, v:0}) [2025-01-08 00:51:09.899178] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:09.899186] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:09.899200] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:09.899272] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:09.899293] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:09.899313] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:09.899335] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:09.899345] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:09.905757] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:09.913140] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15064951064, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:09.916049] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58404][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC1-0-0] [lt=62] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326269994119, keepalive_ts:-1}) [2025-01-08 00:51:09.916096] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58404][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC1-0-0] [lt=46] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326269994119, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:09.917566] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:09.917858] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=94] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:09.920049] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18667070}], scn_array=[cnt:1, {val:1736326269697243001, v:0}], curr_size=122, accum_size=723, time_cost=13908) [2025-01-08 00:51:09.921428] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=25][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365dd90 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44825} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:32120} write_epoch=1, prev=this=0x2ad8c1457368 trans_version={val:1736322641094761000, v:0} scn={val:1736322641094761000, v:0} tx_id={txid:40108} prev=(nil) next=(nil) modify_count=0 acc_checksum=1277335154 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269797549000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:8294} write_epoch=1, snapshot_version={val:1736326269911989000, v:0}, *this={this=0x2ad8c14572d8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1457368 latest_compact_node=(nil) max_trans_version={val:1736326208856817000, v:0} max_trans_id=44742 max_elr_trans_version={val:1736326208856817000, v:0} max_elr_trans_id=44742 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641094761000, v:0} min_modify_scn={val:1736322641094761000, v:0}}) [2025-01-08 00:51:09.921484] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=54][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365dd90 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44825} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:32120} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326299804154, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326299804154, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326269911989000, v:0}, tx_id:{txid:44825}, scn:{branch:0, seq:32117}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44825}, tx_desc:{this:0x2ad8b09c9da0, tx_id:{txid:44825}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326269920732, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326269974341, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a4550, mem_ctx:0x2ad89c0a5ad0, tx_scn:{branch:0, seq:32120}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:09.921574] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=88][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44825}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:32120}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:09.921591] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44825} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:09.921623] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:09.921634] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: C42588F62D2B0600, int: 1736326269904324}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: C42588F62D2B0600, int: 1736326269904324}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 01BCF90000000000, int: 16366593, num_digit0: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00C0720100000000, int: 24297472}]}) [2025-01-08 00:51:09.921723] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:50) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=55] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=5531, insert_dup_release_count=5531, col_map_alloc_count=0, col_map_release_count=0) [2025-01-08 00:51:09.921749] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=16][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:09.932351] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A96-0-0] [lt=31] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:09.942046] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4015] clock out of order(cur_ts=1736326269942042, cur_ts_=1736326269920732, delta=21310) [2025-01-08 00:51:09.942137] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=107][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2162438}], scn_array=[cnt:1, {val:1736326269697243001, v:0}], curr_size=122, accum_size=478, time_cost=25183) [2025-01-08 00:51:09.944601] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=49] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:41799}, to_seq_no={branch:0, seq:32118}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44825} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:09.944703] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:152) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=60] transaction rpc statistics(total_rt=3749, total_process=12, avg_rt=288) [2025-01-08 00:51:09.949098] INFO [STORAGE.TRANS] rollback_to_savepoint (ob_trans_part_ctx.cpp:8738) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=17] rollback succeed trivially(trans_id={txid:44825}, ls_id={id:1}, op_sn=3, to_scn={branch:0, seq:32118}, last_scn={branch:0, seq:32118}) [2025-01-08 00:51:09.954154] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=26] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12811018240, fast_delete_log_mode=false) [2025-01-08 00:51:09.958694] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=10] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:09.969525] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=43] [APPLY STAT CB APPEND COST TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=129, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:09.969576] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=49] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=114674, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:09.969587] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:09.969647] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=59] [APPLY STAT CB EXECUTE TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=60136, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:09.969656] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] [APPLY STAT CB TOTAL TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=174940, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:09.970008] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2484) [46144][pnio1][T1][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=26][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:44825}, sender_ls_id={id:1}, receiver_ls_id={id:9223372036854775807}, msg_type=67, status=-4018, receiver_addr="192.168.159.139:2882", request_id=1736322504776623) [2025-01-08 00:51:09.970067] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:288) [46144][pnio1][T0][YB42C0A89F8B-00062B2D174BB84A-0-0] [lt=56][errcode=0] trx rpc callback(ret=0, status=-4018, dst="192.168.159.139:2882", result={status:-4018, send_timestamp:1736326269949133, private_data:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:09.982456] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84B-0-0] [lt=13] get number(ret=0, service_type_=0, range=1, base_id=1736326269979940000, start_id=1736326269979940000, end_id=1736326269979940001) [2025-01-08 00:51:10.006278] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.009200] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=111] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:10.015903] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46493][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84C-0-0] [lt=42][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365e190 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44826} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:17383} write_epoch=1, prev=this=0x2ad8dae1b7e8 trans_version={val:1736322641115540001, v:0} scn={val:1736322641115540001, v:0} tx_id={txid:40109} prev=(nil) next=(nil) modify_count=0 acc_checksum=2685977296 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:10458} write_epoch=1, snapshot_version={val:1736326270007358000, v:0}, *this={this=0x2ad8dae1b758 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae1b7e8 latest_compact_node=(nil) max_trans_version={val:1736326208900106000, v:0} max_trans_id=44743 max_elr_trans_version={val:1736326208900106000, v:0} max_elr_trans_id=44743 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641115540001, v:0} min_modify_scn={val:1736322641115540001, v:0}}) [2025-01-08 00:51:10.016046] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46493][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84C-0-0] [lt=140][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365e190 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44826} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:17383} write_epoch=1, ctx={this:0x2ad8b7951678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326299898786, tablet_id:{id:309}, table_iter:0x2ad8b794fb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326299898786, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270007358000, v:0}, tx_id:{txid:44826}, scn:{branch:0, seq:17380}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44826}, tx_desc:{this:0x2ad8bc152780, tx_id:{txid:44826}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270015178, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270079999, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c060d50, mem_ctx:0x2ad89c0622d0, tx_scn:{branch:0, seq:17383}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.016142] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46493][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84C-0-0] [lt=95][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44826}, data:0x2ad8b794e508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:17383}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.016161] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46493][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84C-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44826} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.016277] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46493][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84C-0-0] [lt=113][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.016292] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46493][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84C-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96920c6a0, local_datums:0x2ad96920c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96920c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96920c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96920c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96920c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad96920c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96920c375, hex: 3C9A89F62D2B0600, int: 1736326269999676}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96920c37d, hex: 3C9A89F62D2B0600, int: 1736326269999676}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96920c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96920c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.016402] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46493][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84C-0-0] [lt=58][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.027316] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C632-0-0] [lt=33][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8da4c3110 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55694} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:16350} write_epoch=1, prev=this=0x2ad8c146c068 trans_version={val:1736322701346044000, v:0} scn={val:1736322701346044000, v:0} tx_id={txid:50196} prev=0x2ad8c146beb0 next=(nil) modify_count=2 acc_checksum=1214755906 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:18} write_epoch=3, snapshot_version={val:1736326270023933000, v:0}, *this={this=0x2ad8c14574c8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad8c146c068 latest_compact_node=(nil) max_trans_version={val:1736326208002382000, v:0} max_trans_id=55588 max_elr_trans_version={val:1736326208002382000, v:0} max_elr_trans_id=55588 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=3 max_modify_scn={val:1736322701346044000, v:0} min_modify_scn={val:1736322641103233000, v:0}}) [2025-01-08 00:51:10.027406] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C632-0-0] [lt=88][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8da4c3110 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55694} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:16350} write_epoch=1, ctx={this:0x2ad8b7855678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326299920562, tablet_id:{id:309}, table_iter:0x2ad8b7853b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326299920562, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270023933000, v:0}, tx_id:{txid:55694}, scn:{branch:0, seq:16347}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55694}, tx_desc:{this:0x2ad8b94558a0, tx_id:{txid:55694}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270026576, commit_ts:-1, finish_ts:-1, timeout_us:29999978, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270101242, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0191d0, mem_ctx:0x2ad89c01a750, tx_scn:{branch:0, seq:16350}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.027538] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C632-0-0] [lt=129][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55694}, data:0x2ad8b7852508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:16350}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.027566] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C632-0-0] [lt=26][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55694} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.027624] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C632-0-0] [lt=54][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.027642] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C632-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96a00c6a0, local_datums:0x2ad96a00c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96a00c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96a00c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c373, hex: 40538BF62D2B0600, int: 1736326270112576}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c37b, hex: 40538BF62D2B0600, int: 1736326270112576}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c383, hex: A16FD80100000000, int: 30961569}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c38b, hex: 0000730200000000, int: 41091072}]}) [2025-01-08 00:51:10.027794] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C632-0-0] [lt=86][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.027991] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84C-0-0] [lt=19] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:37594}, to_seq_no={branch:0, seq:17381}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44826} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.031642] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C632-0-0] [lt=36] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:21954}, to_seq_no={branch:0, seq:16348}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55694} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.052733] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=33] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:10.054835] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46380][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C633-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326270052653000, start_id=1736326270052653000, end_id=1736326270052653001) [2025-01-08 00:51:10.071954] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4559-0-0] [lt=35][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18043, log_task={header:{begin_lsn:{lsn:2196093}, end_lsn:{lsn:2196215}, log_id:18043, min_scn:{val:1736326269649674003, v:0}, max_scn:{val:1736326269649674003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2195971}, prev_proposal_id:1, committed_end_lsn:{lsn:2196093}, data_checksum:1715074492, accum_checksum:2359832622, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326269963260, freeze_ts:1736326269963260, submit_ts:1736326269963273, flushed_ts:1736326269970307, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:7034}, fs_cb_begin_ts=1736326270071848, log_life_time=108588) [2025-01-08 00:51:10.074562] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=217][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84254401}], scn_array=[cnt:1, {val:1736326270052653000, v:0}], curr_size=560, accum_size=6284, time_cost=17533) [50560]OB_MALLOC COST TOO MUCH TIME, cost_time=281010, tenant_id=1001, label=CoStack, ctx_id=8, prio=1, size=515136, owner: ObMalloc, click_count: 1, time dist:[ALLOC_CHUNK_END=281005], seq:[0] [2025-01-08 00:51:10.087743] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dfc030, session_id:3221581808, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326269300791, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:10.087853] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=97] kill and revert session(conn.sessid_=3221581808, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:51:10.087869] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] sock ref clean, do destroy(*s={this:0x2ad8a0dfc030, session_id:3221581808, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326269300791, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:10.087932] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221581808, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:51:10.087972] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=37] connection close(sessid=3221581808, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AAB-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:4597046, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:4597047, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:4597048, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:4597048, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:4597048, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:4597048, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:4597048, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:4597048, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:4597048, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:10.088260] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=47] free session successfully(sessid=3221581808) [2025-01-08 00:51:10.088296] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=33] mark session id unused(sessid=3221581808) [2025-01-08 00:51:10.088865] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=144] recycle one thread(this=0x5572d527f900, total=6, remain=5) [2025-01-08 00:51:10.088902] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=34] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15065945385, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:10.088921] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7788783349, tenant_id:1002}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:10.103006] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC37F7-0-0] [lt=45][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17938, log_task={header:{begin_lsn:{lsn:2183319}, end_lsn:{lsn:2183441}, log_id:17938, min_scn:{val:1736326269768584003, v:0}, max_scn:{val:1736326269768584003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2183197}, prev_proposal_id:1, committed_end_lsn:{lsn:2183319}, data_checksum:940009244, accum_checksum:4028030369, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326269999916, freeze_ts:1736326269999916, submit_ts:1736326269999928, flushed_ts:1736326270015291, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:15363}, fs_cb_begin_ts=1736326270102993, log_life_time=103077) [2025-01-08 00:51:10.103511] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C634-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8da4c32e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55695} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:23025} write_epoch=1, prev=this=0x2ad8c1457750 trans_version={val:1736322641116642000, v:0} scn={val:1736322641116642000, v:0} tx_id={txid:50112} prev=(nil) next=(nil) modify_count=0 acc_checksum=1602798291 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:11449} write_epoch=1, snapshot_version={val:1736326270086166000, v:0}, *this={this=0x2ad8c14576c0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1457750 latest_compact_node=(nil) max_trans_version={val:1736326208105527000, v:0} max_trans_id=55589 max_elr_trans_version={val:1736326208105527000, v:0} max_elr_trans_id=55589 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641116642000, v:0} min_modify_scn={val:1736322641116642000, v:0}}) [2025-01-08 00:51:10.103566] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C634-0-0] [lt=54][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8da4c32e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55695} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:23025} write_epoch=1, ctx={this:0x2ad8b7855678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326299993917, tablet_id:{id:309}, table_iter:0x2ad8b7853b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326299993917, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270086166000, v:0}, tx_id:{txid:55695}, scn:{branch:0, seq:15037}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55695}, tx_desc:{this:0x2ad8b94cae60, tx_id:{txid:55695}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270102479, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270169019, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0191d0, mem_ctx:0x2ad89c01a750, tx_scn:{branch:0, seq:23025}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.103749] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C634-0-0] [lt=179][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55695}, data:0x2ad8b7852508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:23025}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.103812] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C634-0-0] [lt=62][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55695} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.103851] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C634-0-0] [lt=36][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.103873] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=21] add async task(this=task_type:N9oceanbase10rootserver21ObPurgeRecyclebinTaskE, timeout_check:False) [2025-01-08 00:51:10.103863] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C634-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96a00c6a0, local_datums:0x2ad96a00c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96a00c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad96a00c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c375, hex: 8B4B8CF62D2B0600, int: 1736326270176139}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c37d, hex: 8B4B8CF62D2B0600, int: 1736326270176139}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.103971] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C634-0-0] [lt=59][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.104105] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=55] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:10.107155] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=7][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365e4b8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44827} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:5310} write_epoch=1, prev=this=0x2ad8c1563780 trans_version={val:1736324386058996000, v:0} scn={val:1736324386058996000, v:0} tx_id={txid:42211} prev=(nil) next=(nil) modify_count=0 acc_checksum=3597167689 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:12192} write_epoch=1, snapshot_version={val:1736326270104454000, v:0}, *this={this=0x2ad8c15636f0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1563780 latest_compact_node=(nil) max_trans_version={val:1736326208973773000, v:0} max_trans_id=44744 max_elr_trans_version={val:1736326208973773000, v:0} max_elr_trans_id=44744 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386058996000, v:0} min_modify_scn={val:1736324386058996000, v:0}}) [2025-01-08 00:51:10.107327] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=170][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365e4b8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44827} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:5310} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300003943, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300003943, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270104454000, v:0}, tx_id:{txid:44827}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44827}, tx_desc:{this:0x2ad8b09d5fe0, tx_id:{txid:44827}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270105662, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270192042, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:5310}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.107650] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=321][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44827}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:5310}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.107669] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44827} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.107699] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.107710] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c372, hex: 043A8BF62D2B0600, int: 1736326270106116}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37a, hex: 043A8BF62D2B0600, int: 1736326270106116}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.108021] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=201][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.108877] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46369][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C634-0-0] [lt=97] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:28334}, to_seq_no={branch:0, seq:15038}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55695} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.110436] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=61] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:10.112838] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:10.113962] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:10.114011] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.114563] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46502][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=5] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:11546}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44827} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.120563] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:10.120898] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:10.122857] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=45] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=660597, clean_num=31457) [2025-01-08 00:51:10.125572] INFO [STORAGE.TRANS] rollback_to_savepoint (ob_trans_part_ctx.cpp:8738) [46508][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=29] rollback succeed trivially(trans_id={txid:44827}, ls_id={id:1}, op_sn=3, to_scn={branch:0, seq:5}, last_scn={branch:0, seq:5}) [2025-01-08 00:51:10.127802] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18667314}], scn_array=[cnt:1, {val:1736326269697243003, v:0}], curr_size=122, accum_size=967, time_cost=12755) [2025-01-08 00:51:10.132508] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2484) [46144][pnio1][T1][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=15][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:44827}, sender_ls_id={id:1}, receiver_ls_id={id:9223372036854775807}, msg_type=67, status=-4018, receiver_addr="192.168.159.139:2882", request_id=1736322504776625) [2025-01-08 00:51:10.132557] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:288) [46144][pnio1][T0][YB42C0A89F8B-00062B2D174BB84E-0-0] [lt=46][errcode=0] trx rpc callback(ret=0, status=-4018, dst="192.168.159.139:2882", result={status:-4018, send_timestamp:1736326270125628, private_data:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:10.138629] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C635-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326270137671000, start_id=1736326270137671000, end_id=1736326270137671001) [46262]OB_MALLOC COST TOO MUCH TIME, cost_time=334019, tenant_id=1001, label=CoStack, ctx_id=8, prio=1, size=515136, owner: ObMalloc, click_count: 3, time dist:[ObMalloc#start=4, ObMalloc#end=4, ALLOC_CHUNK_END=334008], seq:[0, 1, 2] [2025-01-08 00:51:10.145014] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] recycle one thread(this=0x2ad8c0eb0e70, total=7, remain=6) [2025-01-08 00:51:10.145113] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=96] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7789114820, tenant_id:1002}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:10.145862] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=43] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:14993626753, tenant_id:1}, cur_thread_count=6, cnt=1, new_thread_count=7) [2025-01-08 00:51:10.152721] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=135] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:7754751096, tenant_id:1001}, cur_thread_count=7, cnt=1, new_thread_count=8) [2025-01-08 00:51:10.152777] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=50] success to update min reserved snapshot(reserved_snapshot=1736324464299814000, duration=1800, snapshot_gc_ts=1736326264299814000) [2025-01-08 00:51:10.153932] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C636-0-0] [lt=76][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365e7e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55696} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4401} write_epoch=1, prev=this=0x2ad8c1563fa8 trans_version={val:1736324386193967000, v:0} scn={val:1736324386193967000, v:0} tx_id={txid:52596} prev=(nil) next=(nil) modify_count=0 acc_checksum=3372950423 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:33080} write_epoch=1, snapshot_version={val:1736326270149469000, v:0}, *this={this=0x2ad8c1563f18 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1563fa8 latest_compact_node=(nil) max_trans_version={val:1736326208156081000, v:0} max_trans_id=55594 max_elr_trans_version={val:1736326208156081000, v:0} max_elr_trans_id=55594 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386193967000, v:0} min_modify_scn={val:1736324386193967000, v:0}}) [2025-01-08 00:51:10.154004] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C636-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365e7e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55696} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4401} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300050632, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300050632, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270149469000, v:0}, tx_id:{txid:55696}, scn:{branch:0, seq:3222}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55696}, tx_desc:{this:0x2ad8b09e01c0, tx_id:{txid:55696}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270151831, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270240552, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:4401}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.154134] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C636-0-0] [lt=128][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55696}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4401}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.154333] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C636-0-0] [lt=196][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55696} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.154366] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C636-0-0] [lt=31][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.154376] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C636-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c372, hex: 464C8DF62D2B0600, int: 1736326270241862}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37a, hex: 464C8DF62D2B0600, int: 1736326270241862}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.154508] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C636-0-0] [lt=85][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.156397] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C636-0-0] [lt=38] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7989}, to_seq_no={branch:0, seq:4399}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55696} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.161883] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58434][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:10.161983] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91] worker thread created(tenant_->id()=1001, group_id_=1, token=4) [2025-01-08 00:51:10.166065] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB850-0-0] [lt=33][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365e9a8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44828} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:15664} write_epoch=1, prev=this=0x2ad8c1563968 trans_version={val:1736324386116687000, v:0} scn={val:1736324386116687000, v:0} tx_id={txid:42212} prev=(nil) next=(nil) modify_count=0 acc_checksum=1789153443 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4159} write_epoch=1, snapshot_version={val:1736326270154707000, v:0}, *this={this=0x2ad8c15638d8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1563968 latest_compact_node=(nil) max_trans_version={val:1736326209026978000, v:0} max_trans_id=44745 max_elr_trans_version={val:1736326209026978000, v:0} max_elr_trans_id=44745 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386116687000, v:0} min_modify_scn={val:1736324386116687000, v:0}}) [2025-01-08 00:51:10.166252] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB850-0-0] [lt=185][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365e9a8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44828} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:15664} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300054382, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300054382, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270154707000, v:0}, tx_id:{txid:44828}, scn:{branch:0, seq:9148}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44828}, tx_desc:{this:0x2ad8b09de160, tx_id:{txid:44828}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270164796, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270240554, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:15664}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.166346] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB850-0-0] [lt=93][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44828}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:15664}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.166362] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB850-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44828} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.166413] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB850-0-0] [lt=48][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.166427] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB850-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c372, hex: 2BFA8BF62D2B0600, int: 1736326270155307}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37a, hex: 2BFA8BF62D2B0600, int: 1736326270155307}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.166723] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB850-0-0] [lt=251][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.168609] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=14] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324464299814000) [2025-01-08 00:51:10.168649] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=36] update reserved snapshot finished(cost_ts=15875, reserved_snapshot=1736324464299814000) [2025-01-08 00:51:10.173010] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB850-0-0] [lt=41] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:15669}, to_seq_no={branch:0, seq:9149}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44828} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.179070] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A53-0-0] [lt=39] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:10.196269] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB852-0-0] [lt=16][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365ecd0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44829} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=1, prev=this=0x2ad8dae89758 trans_version={val:1736326209120785000, v:0} scn={val:1736326209120785000, v:0} tx_id={txid:44746} prev=0x2ad963656960 next=(nil) modify_count=118 acc_checksum=880948073 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:56843} write_epoch=3, snapshot_version={val:1736326270193349000, v:0}, *this={this=0x2ad8c14578a8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8dae89758 latest_compact_node=0x2ad8dae87218 max_trans_version={val:1736326209120785000, v:0} max_trans_id=44746 max_elr_trans_version={val:1736326209120785000, v:0} max_elr_trans_id=44746 latest_compact_ts=1736326087619820 last_compact_cnt=113 total_trans_node_cnt=119 max_modify_scn={val:1736326209120785000, v:0} min_modify_scn={val:1736322641129989000, v:0}}) [2025-01-08 00:51:10.196365] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB852-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365ecd0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44829} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300093741, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300093741, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270193349000, v:0}, tx_id:{txid:44829}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44829}, tx_desc:{this:0x2ad8b09ec400, tx_id:{txid:44829}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270195590, commit_ts:-1, finish_ts:-1, timeout_us:29999974, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270272800, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:7}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.196463] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB852-0-0] [lt=96][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44829}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:7}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.196480] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB852-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44829} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.196512] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB852-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.196524] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB852-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: D2998CF62D2B0600, int: 1736326270196178}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: D2998CF62D2B0600, int: 1736326270196178}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 92B5050900000000, int: 151369106}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 92B5050900000000, int: 151369106}]}) [2025-01-08 00:51:10.196625] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB852-0-0] [lt=58][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.198583] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB852-0-0] [lt=99] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44829} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.214092] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.214899] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A54-0-0] [lt=303] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:10.222845] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58434][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2C6F-0-0] [lt=31] init point thread id with(&point=0x5572d50e39c0, idx_=1090, point=[thread id=58434, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58434) [2025-01-08 00:51:10.223342] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=10] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:10.223430] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=80] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:10.223446] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=13] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:10.223469] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=21] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:10.228356] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A55-0-0] [lt=47] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:10.228803] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=43] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:10.231006] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=42] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:10.232036] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A56-0-0] [lt=75] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:10.233145] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=45] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:10.233185] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=38] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:10.233094] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C638-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365f128 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55697} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:3286} write_epoch=1, prev=this=0x2ad8c1564380 trans_version={val:1736324386214262000, v:0} scn={val:1736324386214262000, v:0} tx_id={txid:52597} prev=(nil) next=(nil) modify_count=0 acc_checksum=1313669344 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:3643} write_epoch=1, snapshot_version={val:1736326270200985000, v:0}, *this={this=0x2ad8c15642f0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1564380 latest_compact_node=(nil) max_trans_version={val:1736326208201067000, v:0} max_trans_id=55596 max_elr_trans_version={val:1736326208201067000, v:0} max_elr_trans_id=55596 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386214262000, v:0} min_modify_scn={val:1736324386214262000, v:0}}) [2025-01-08 00:51:10.233294] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C638-0-0] [lt=198][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365f128 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55697} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:3286} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300100467, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300100467, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270200985000, v:0}, tx_id:{txid:55697}, scn:{branch:0, seq:3283}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55697}, tx_desc:{this:0x2ad8b09f8640, tx_id:{txid:55697}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270232693, commit_ts:-1, finish_ts:-1, timeout_us:29999967, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270289873, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:3286}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.233412] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C638-0-0] [lt=116][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55697}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:3286}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.233429] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C638-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55697} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.233460] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C638-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.233474] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C638-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c372, hex: D2078EF62D2B0600, int: 1736326270289874}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37a, hex: D2078EF62D2B0600, int: 1736326270289874}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.233681] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C638-0-0] [lt=113][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.234931] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084878-0-0] [lt=63] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:10.235029] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084878-0-0] [lt=50] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:10.235045] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084878-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:10.238398] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A57-0-0] [lt=21] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:10.238620] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2162804}], scn_array=[cnt:1, {val:1736326270112410000, v:0}], curr_size=122, accum_size=844, time_cost=26622) [2025-01-08 00:51:10.239435] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB854-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365f2f0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44830} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:10} write_epoch=1, prev=this=0x2ad8c1564190 trans_version={val:1736324386214262001, v:0} scn={val:1736324386214262001, v:0} tx_id={txid:42214} prev=(nil) next=(nil) modify_count=0 acc_checksum=3172668830 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3645} write_epoch=1, snapshot_version={val:1736326270226356000, v:0}, *this={this=0x2ad8c1564100 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1564190 latest_compact_node=(nil) max_trans_version={val:1736326209173627000, v:0} max_trans_id=44747 max_elr_trans_version={val:1736326209173627000, v:0} max_elr_trans_id=44747 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386214262001, v:0} min_modify_scn={val:1736324386214262001, v:0}}) [2025-01-08 00:51:10.239485] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB854-0-0] [lt=49][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365f2f0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44830} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:10} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300125379, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300125379, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270226356000, v:0}, tx_id:{txid:44830}, scn:{branch:0, seq:6}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44830}, tx_desc:{this:0x2ad8b09fa6a0, tx_id:{txid:44830}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270238585, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270293152, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:10}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.239586] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB854-0-0] [lt=100][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44830}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:10}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.239602] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB854-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44830} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.239630] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB854-0-0] [lt=26][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.239710] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB854-0-0] [lt=76][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: EF138DF62D2B0600, int: 1736326270227439}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: EF138DF62D2B0600, int: 1736326270227439}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 710B080000000000, int: 527217}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00D03D0000000000, int: 4050944}]}) [2025-01-08 00:51:10.239793] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB854-0-0] [lt=51][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.240218] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD6-0-0] [lt=124] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:10.240263] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD6-0-0] [lt=37] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=48) [2025-01-08 00:51:10.241534] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB854-0-0] [lt=64] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:13}, to_seq_no={branch:0, seq:7}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44830} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.242832] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=13] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:10.245479] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=43] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:10.246989] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=66] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:10.247037] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=47] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:10.249345] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084878-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:10.249464] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084878-0-0] [lt=39] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:10.249485] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084878-0-0] [lt=19] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:10.253670] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=28] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:10.254309] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:10.256000] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=40] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:10.258323] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=88] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:10.258360] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=36] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:10.260469] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084878-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:10.260560] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084878-0-0] [lt=39] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:10.260578] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084878-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:10.261230] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1002}, {id:1003}]) [2025-01-08 00:51:10.260600] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=12] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:10.261329] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=725] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:10.261380] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=43] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:10.261391] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:10.261434] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084878-0-0] [lt=38] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:10.261458] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084878-0-0] [lt=23] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:10.261470] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=7] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:10.261477] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:10.261560] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084878-0-0] [lt=79] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:10.261596] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084878-0-0] [lt=35] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:10.261611] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=9] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:10.261617] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:10.261636] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084878-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:10.261649] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084878-0-0] [lt=13] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:10.265642] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:10.265735] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:10.265798] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=0] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:10.265824] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=21] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=310, cache_node_num=310) [2025-01-08 00:51:10.265835] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=310, cache_node_num=310) [2025-01-08 00:51:10.265845] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=8] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:10.266498] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=32] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:10.266526] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=26] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:10.266536] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=9] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:10.266565] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:10.266579] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=13] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:10.266588] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:10.266600] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:10.266616] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084878-0-0] [lt=15] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:10.268778] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=310, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}]) [2025-01-08 00:51:10.269684] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=901] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:10.278882] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3A1-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:10.280797] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB856-0-0] [lt=74][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365f570 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44831} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:10} write_epoch=1, prev=this=0x2ad8c1573950 trans_version={val:1736324508824146000, v:0} scn={val:1736324508824146000, v:0} tx_id={txid:42418} prev=0x2ad8c1573798 next=(nil) modify_count=4 acc_checksum=3409534941 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:18565} write_epoch=3, snapshot_version={val:1736326270274087000, v:0}, *this={this=0x2ad8c15646a8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8c1573950 latest_compact_node=(nil) max_trans_version={val:1736326209214264000, v:0} max_trans_id=44748 max_elr_trans_version={val:1736326209214264000, v:0} max_elr_trans_id=44748 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324508824146000, v:0} min_modify_scn={val:1736324386269257000, v:0}}) [2025-01-08 00:51:10.280878] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB856-0-0] [lt=79][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365f570 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44831} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:10} write_epoch=1, ctx={this:0x2ad8ad3cf678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300176473, tablet_id:{id:309}, table_iter:0x2ad8ad3cdb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300176473, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270274087000, v:0}, tx_id:{txid:44831}, scn:{branch:0, seq:5}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44831}, tx_desc:{this:0x2ad8b8b3e3c0, tx_id:{txid:44831}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270279162, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270293180, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:10}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.281001] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB856-0-0] [lt=120][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44831}, data:0x2ad8ad3cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:10}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.281146] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB856-0-0] [lt=143][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44831} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.281235] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB856-0-0] [lt=85][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.281250] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB856-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96500c6a0, local_datums:0x2ad96500c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96500c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96500c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96500c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96500c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96500c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96500c373, hex: 27CF8DF62D2B0600, int: 1736326270275367}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96500c37b, hex: 27CF8DF62D2B0600, int: 1736326270275367}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96500c383, hex: 1ED9080000000000, int: 579870}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96500c38b, hex: 00802F0000000000, int: 3112960}]}) [2025-01-08 00:51:10.281374] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB856-0-0] [lt=75][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.282298] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:10.282459] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=90] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:10.284876] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB856-0-0] [lt=55] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:13}, to_seq_no={branch:0, seq:6}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44831} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.289417] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=32] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:10.302638] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A58-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:10.302684] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46388][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C638-0-0] [lt=395] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:101236}, to_seq_no={branch:0, seq:3284}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55697} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.303530] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A914B-0-0] [lt=57] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=844) [2025-01-08 00:51:10.304098] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:10.304348] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:10.307685] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB858-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365f7f8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44832} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:2742} write_epoch=1, prev=this=0x2ad8c15649e0 trans_version={val:1736324386317997000, v:0} scn={val:1736324386317997000, v:0} tx_id={txid:42216} prev=(nil) next=(nil) modify_count=0 acc_checksum=866976763 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:8267} write_epoch=1, snapshot_version={val:1736326270303329000, v:0}, *this={this=0x2ad8c1564950 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c15649e0 latest_compact_node=(nil) max_trans_version={val:1736326209252864000, v:0} max_trans_id=44749 max_elr_trans_version={val:1736326209252864000, v:0} max_elr_trans_id=44749 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386317997000, v:0} min_modify_scn={val:1736324386317997000, v:0}}) [2025-01-08 00:51:10.307758] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB858-0-0] [lt=71][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365f7f8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44832} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:2742} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300205119, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300205119, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270303329000, v:0}, tx_id:{txid:44832}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44832}, tx_desc:{this:0x2ad8b0806a00, tx_id:{txid:44832}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270306990, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270393925, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:2742}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.307858] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB858-0-0] [lt=98][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44832}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:2742}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.307876] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB858-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44832} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.307924] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB858-0-0] [lt=45][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.307939] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB858-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c375, hex: 56418EF62D2B0600, int: 1736326270304598}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37d, hex: 56418EF62D2B0600, int: 1736326270304598}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.308109] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB858-0-0] [lt=88][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.309449] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=55] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=97152, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:10.309478] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1333) [2025-01-08 00:51:10.309566] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=87306, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:10.309841] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=421] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=108, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:10.309979] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB858-0-0] [lt=60] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:2745}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44832} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.311199] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:10.313355] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=46] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:10.313388] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-01-08 00:51:10.313399] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:10.313412] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:10.313425] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193774226676, flush_io_finished_ret_:0, flush_io_finished_round_:1933, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:10.314672] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.315162] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58429][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:10.315475] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=93] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=51922, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:10.315498] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=966) [2025-01-08 00:51:10.315515] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=2205, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:10.315543] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=74, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:10.316373] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8517-0-0] [lt=41] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=35, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:10.316406] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8517-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=59148, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:10.316569] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1201, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:10.318655] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8516-0-0] [lt=75] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:51:10.318714] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8516-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=36, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:10.318731] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8516-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=201932, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:10.318739] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8516-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=26163, avg_log_submit_to_flush_time=148331, avg_log_submit_to_slide_time=175768) [2025-01-08 00:51:10.319445] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=60] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.319547] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A59-0-0] [lt=78] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:10.319601] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1333, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:10.319626] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17954, miss_cnt:0, cache_read_size:2196572, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:10.319692] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.321009] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=77] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:10.323082] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11786) [2025-01-08 00:51:10.323480] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:10.328703] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1736326270328700, bytes: 871616860, bw: 0.664040 MB/s, add_ts: 1004788, add_bytes: 699630 [2025-01-08 00:51:10.330784] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85A-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365f9d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44833} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:7} write_epoch=1, prev=this=0x2ad8c1564bd8 trans_version={val:1736324386348542000, v:0} scn={val:1736324386348542000, v:0} tx_id={txid:42217} prev=(nil) next=(nil) modify_count=0 acc_checksum=836285912 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:3791} write_epoch=1, snapshot_version={val:1736326270328020000, v:0}, *this={this=0x2ad8c1564b48 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1564bd8 latest_compact_node=(nil) max_trans_version={val:1736326209294125000, v:0} max_trans_id=44750 max_elr_trans_version={val:1736326209294125000, v:0} max_elr_trans_id=44750 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386348542000, v:0} min_modify_scn={val:1736324386348542000, v:0}}) [2025-01-08 00:51:10.330857] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85A-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365f9d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44833} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:7} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300228016, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300228016, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270328020000, v:0}, tx_id:{txid:44833}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44833}, tx_desc:{this:0x2ad8b0812c40, tx_id:{txid:44833}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270330373, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270410562, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:7}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.330972] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85A-0-0] [lt=113][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44833}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:7}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.330990] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85A-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44833} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.331020] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85A-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.331030] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85A-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c375, hex: 4BA68EF62D2B0600, int: 1736326270330443}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37d, hex: 4BA68EF62D2B0600, int: 1736326270330443}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.331127] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85A-0-0] [lt=54][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.332311] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB85A-0-0] [lt=42] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44833} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.334054] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A5A-0-0] [lt=20] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:10.335424] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAC-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:10.335452] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAC-0-0] [lt=26] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:10.335658] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=37] failed to get result(ret=-4008) [2025-01-08 00:51:10.335812] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=38] check new jobs(ret=0, tenant_id=1001, is_oracle_tenant=false, job_infos=[{tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326185605925, start_date_:1736325283345722, last_date_:1736326183353587, this_date_:0, next_date_:1736327083345722, end_date_:64060560000000000, total_:6596354, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736410783345628, last_date_:0, this_date_:0, next_date_:1736410783345628, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:51:10.336023] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=207] check all tenants(ret=0) [2025-01-08 00:51:10.341184] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A5C-0-0] [lt=11] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:10.349307] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:10.349392] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:10.352237] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] get_max_applied_scn(scn={val:1736326270112410000, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2163048}, palf_committed_end_scn_:{val:1736326270112410001, v:0}, last_check_scn_:{val:1736326270112410001, v:0}, max_applied_cb_scn_:{val:1736326270112410000, v:0}}) [2025-01-08 00:51:10.352340] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=99] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2163048}, readable_log_tail:{lsn:2163048}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64941720, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:10.358642] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=10] current monitor number(seq_=-1) [2025-01-08 00:51:10.360716] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185421-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:10.366985] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=13] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:10.382121] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2183863}, scn={val:1736326270256872001, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:10.382183] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=51] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=23, stat_interval=1000000, avg (count/cost)=206, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:10.382202] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=18] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=23, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:10.382218] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=10] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:31, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326270381990, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2183919}, next_to_submit_scn_:{val:1736326270256872002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2183807}, end_lsn:{lsn:2183919}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2183919}, readable_log_tail:{lsn:2183919}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64920849, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326270256872001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326270256872001, v:0}, data_checksum:2445183700, flag:37613}}, init_mode_version:0, accumulate_checksum:1852768066, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326270256872001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17852, miss_cnt:0, cache_read_size:2183919, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2183863}, scn_:{val:1736326270256872001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326270378441, first_handle_ts_:1736326270382216, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad962c04150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:51:10.382917] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324465595928000) [2025-01-08 00:51:10.382947] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2183863}, scn_:{val:1736326270256872001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326270378441, first_handle_ts_:1736326270382855, replay_cost_:90, retry_cost_:-1, read_log_buf_:0x2ad962c04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326270381990, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2183919}, next_to_submit_scn_:{val:1736326270256872002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2183807}, end_lsn:{lsn:2183919}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2183919}, readable_log_tail:{lsn:2183919}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64920849, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326270256872001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326270256872001, v:0}, data_checksum:2445183700, flag:37613}}, init_mode_version:0, accumulate_checksum:1852768066, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326270256872001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17852, miss_cnt:0, cache_read_size:2183919, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=9, ret=0) [2025-01-08 00:51:10.384472] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB85B-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=12327) [2025-01-08 00:51:10.392842] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=6] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:10.392891] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=39] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:10.392908] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1) [2025-01-08 00:51:10.405588] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:10.405727] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=126] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:10.405765] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=35] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:10.405776] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:10.405783] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:10.405791] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=99) [2025-01-08 00:51:10.406913] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85E-0-0] [lt=34][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365fd70 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44835} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=1, prev=this=0x2ad963657a90 trans_version={val:1736326209358736000, v:0} scn={val:1736326209358736000, v:0} tx_id={txid:44752} prev=0x2ad8dae89a00 next=(nil) modify_count=60 acc_checksum=406040890 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7902} write_epoch=3, snapshot_version={val:1736326270405250000, v:0}, *this={this=0x2ad8c15652e8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad963657a90 latest_compact_node=0x2ad8dae89a00 max_trans_version={val:1736326209358736000, v:0} max_trans_id=44752 max_elr_trans_version={val:1736326209358736000, v:0} max_elr_trans_id=44752 latest_compact_ts=1736326209367843 last_compact_cnt=59 total_trans_node_cnt=61 max_modify_scn={val:1736326209358736000, v:0} min_modify_scn={val:1736324386409746000, v:0}}) [2025-01-08 00:51:10.406999] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85E-0-0] [lt=85][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365fd70 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44835} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300304938, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300304938, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270405250000, v:0}, tx_id:{txid:44835}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44835}, tx_desc:{this:0x2ad8b09e62e0, tx_id:{txid:44835}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270406426, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270410605, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:7}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.407095] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85E-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44835}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:7}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.407112] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85E-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44835} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.407142] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85E-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.407154] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85E-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: F0CA8FF62D2B0600, int: 1736326270405360}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: F0CA8FF62D2B0600, int: 1736326270405360}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: AFBC920600000000, int: 110279855}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: AFBC920600000000, int: 110279855}]}) [2025-01-08 00:51:10.407401] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB85E-0-0] [lt=135][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.408892] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB85E-0-0] [lt=27] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44835} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.412493] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=48, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:10.412540] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=8, avg_log_batch_cnt=1, total_group_log_size=966, avg_group_log_size=120) [2025-01-08 00:51:10.412560] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=966) [2025-01-08 00:51:10.412573] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:10.414951] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.415524] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:51:10.420150] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1736326270419894000, start_id=1736326270419894000, end_id=1736326270419894001) [2025-01-08 00:51:10.420385] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:10.422049] INFO [LIB] stat (utility.h:1210) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB85F-0-0] [lt=44] [PALF STAT APPEND COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:10.433059] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=30] gc stale ls task succ [2025-01-08 00:51:10.441043] INFO [STORAGE.COMPACTION] add_event_and_diagnose (ob_medium_loop.cpp:203) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84ED6-0-0] [lt=90] finish schedule all tablet merge(merge_version_=1, schedule_stats_={all_ls_weak_read_ts_ready:true, start_timestamp:0}, loop_cnt=0, tenant_merge_finish=true, is_scan_all_tablet_finish=true, schedule_tablet_cnt={schedule_dag_cnt:0, submit_clog_cnt:0, finish_cnt:989, loop_tablet_cnt:989, force_freeze_cnt:0, wait_rs_validate_cnt:0}, time_guard=GET_TABLET=20.24ms|INIT_TABLET_STATUS=91.58ms|SCHEDULE_NEXT_MEDIUM=6.40ms|SCHEDULE_TABLET_EXECUTE=3.43s|) [2025-01-08 00:51:10.441127] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:95) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84ED6-0-0] [lt=79] MediumLoopTask(cost_ts=3545487) [2025-01-08 00:51:10.441222] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50558][T1001_TimerWK0][T1001][YB42C0A89F89-00062B2D1E484E1F-0-0] [lt=22] MediumCheckTask(cost_ts=10) [2025-01-08 00:51:10.455022] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB860-0-0] [lt=28][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8da4c34b0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44836} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3340} write_epoch=1, prev=this=0x2ad963657d18 trans_version={val:1736326209416973000, v:0} scn={val:1736326209416973000, v:0} tx_id={txid:44753} prev=0x2ad8dae89c88 next=(nil) modify_count=60 acc_checksum=3059102391 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:37325} write_epoch=3, snapshot_version={val:1736326270451023000, v:0}, *this={this=0x2ad8c15656d0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad963657d18 latest_compact_node=0x2ad8dae89c88 max_trans_version={val:1736326209416973000, v:0} max_trans_id=44753 max_elr_trans_version={val:1736326209416973000, v:0} max_elr_trans_id=44753 latest_compact_ts=1736326209421423 last_compact_cnt=59 total_trans_node_cnt=61 max_modify_scn={val:1736326209416973000, v:0} min_modify_scn={val:1736324386439933000, v:0}}) [2025-01-08 00:51:10.455100] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB860-0-0] [lt=75][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8da4c34b0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44836} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3340} write_epoch=1, ctx={this:0x2ad8b79cf678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300351796, tablet_id:{id:309}, table_iter:0x2ad8b79cdb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300351796, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270451023000, v:0}, tx_id:{txid:44836}, scn:{branch:0, seq:3335}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44836}, tx_desc:{this:0x2ad8b5c4b6c0, tx_id:{txid:44836}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270453555, commit_ts:-1, finish_ts:-1, timeout_us:29999916, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270451167, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:3340}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.455258] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB860-0-0] [lt=156][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44836}, data:0x2ad8b79cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:3340}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.455278] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB860-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44836} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.455347] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB860-0-0] [lt=66][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.455363] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB860-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8df80c6a0, local_datums:0x2ad8df80c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8df80c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8df80c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8df80c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8df80c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8df80c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8df80c373, hex: E07D90F62D2B0600, int: 1736326270451168}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8df80c37b, hex: E07D90F62D2B0600, int: 1736326270451168}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8df80c383, hex: DEE7801100000000, int: 293660638}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8df80c38b, hex: DEE7801100000000, int: 293660638}]}) [2025-01-08 00:51:10.455459] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB860-0-0] [lt=61][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.457646] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB860-0-0] [lt=25] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8105}, to_seq_no={branch:0, seq:3336}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44836} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.457928] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=6] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:10.465124] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.465156] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.470417] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=175] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:10.470466] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=41] temporary file module use no blocks [2025-01-08 00:51:10.470478] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:10.470492] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:10.470504] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193774384092, flush_io_finished_ret_:0, flush_io_finished_round_:1929, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:10.470573] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=36] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:10.470592] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-01-08 00:51:10.470598] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=4] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:10.470606] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:10.470615] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193774384109, flush_io_finished_ret_:0, flush_io_finished_round_:3757, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:10.483812] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB862-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963660450 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44837} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:2277} write_epoch=1, prev=this=0x2ad8c1457f08 trans_version={val:1736322641150641000, v:0} scn={val:1736322641150641000, v:0} tx_id={txid:40112} prev=(nil) next=(nil) modify_count=0 acc_checksum=221902601 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3015} write_epoch=1, snapshot_version={val:1736326270481381000, v:0}, *this={this=0x2ad8c1457e78 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1457f08 latest_compact_node=(nil) max_trans_version={val:1736326209454454000, v:0} max_trans_id=44754 max_elr_trans_version={val:1736326209454454000, v:0} max_elr_trans_id=44754 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641150641000, v:0} min_modify_scn={val:1736322641150641000, v:0}}) [2025-01-08 00:51:10.483874] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB862-0-0] [lt=60][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963660450 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44837} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:2277} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300382055, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300382055, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270481381000, v:0}, tx_id:{txid:44837}, scn:{branch:0, seq:2274}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44837}, tx_desc:{this:0x2ad8b0829060, tx_id:{txid:44837}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270482511, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270483028, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:2277}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.483979] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB862-0-0] [lt=103][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44837}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:2277}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.483998] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB862-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44837} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.484030] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB862-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.484041] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB862-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c374, hex: 55FA90F62D2B0600, int: 1736326270483029}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37c, hex: 55FA90F62D2B0600, int: 1736326270483029}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c384, hex: 0080420200000000, int: 37912576}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38c, hex: 0080420200000000, int: 37912576}]}) [2025-01-08 00:51:10.484139] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB862-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.485428] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB862-0-0] [lt=45] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:4661}, to_seq_no={branch:0, seq:2275}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44837} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.499714] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D22484649-0-0] [lt=34] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:51:10.504775] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87079-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:10.504856] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:10.504870] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:10.511510] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58384][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D765-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1567) [2025-01-08 00:51:10.511623] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D765-0-0] [lt=73] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:10.511643] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D765-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=47241, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:10.511690] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58384][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D765-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=350, avg_log_submit_to_flush_time=29474, avg_log_submit_to_slide_time=46890) [2025-01-08 00:51:10.511975] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1567, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:10.512514] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=520] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17854, miss_cnt:0, cache_read_size:2184163, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:10.512859] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1333) [2025-01-08 00:51:10.512886] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:10.515380] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.519526] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46500][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=26][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8da4c3680 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44838} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:99523} write_epoch=1, prev=this=0x2ad8c14582e8 trans_version={val:1736322641160272000, v:0} scn={val:1736322641160272000, v:0} tx_id={txid:40113} prev=(nil) next=(nil) modify_count=0 acc_checksum=4054987080 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:1932} write_epoch=1, snapshot_version={val:1736326270510256000, v:0}, *this={this=0x2ad8c1458258 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c14582e8 latest_compact_node=(nil) max_trans_version={val:1736326209494198000, v:0} max_trans_id=44755 max_elr_trans_version={val:1736326209494198000, v:0} max_elr_trans_id=44755 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641160272000, v:0} min_modify_scn={val:1736322641160272000, v:0}}) [2025-01-08 00:51:10.519587] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46500][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=60][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8da4c3680 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44838} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:99523} write_epoch=1, ctx={this:0x2ad8b7cd3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300411873, tablet_id:{id:309}, table_iter:0x2ad8b7cd1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300411873, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270510256000, v:0}, tx_id:{txid:44838}, scn:{branch:0, seq:7919}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44838}, tx_desc:{this:0x2ad8b6e06a00, tx_id:{txid:44838}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270518126, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270509906, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:99523}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.519684] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46500][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44838}, data:0x2ad8b7cd0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:99523}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.519701] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46500][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44838} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.519732] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46500][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.519744] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46500][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad91dc0c6a0, local_datums:0x2ad91dc0c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad91dc0c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad91dc0c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad91dc0c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad91dc0c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad91dc0c363, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad91dc0c373, hex: 536391F62D2B0600, int: 1736326270509907}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad91dc0c37b, hex: 536391F62D2B0600, int: 1736326270509907}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad91dc0c383, hex: 0050020000000000, int: 151552}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad91dc0c38b, hex: 0050020000000000, int: 151552}]}) [2025-01-08 00:51:10.519851] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46500][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.521263] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=39] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.523505] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:10.524257] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:10.534044] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=291] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:10.534111] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:23, bw:110362, [delay/us]:prepare:11, schedule:47542, submit:161, rt:3557, total:51273) [2025-01-08 00:51:10.534131] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=6, iops=6, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:10.534326] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=178] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:24, avg_bw:110360, [delay/us]: prepare:11, schedule:47542, submit:161, device:3557, total:51273) [2025-01-08 00:51:10.534375] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:10.534412] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:456, bw:1868226, [delay/us]:prepare:5, schedule:2407, submit:2, rt:736, total:3152) [2025-01-08 00:51:10.534433] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:16, bw:74886, [delay/us]:prepare:16, schedule:770, submit:55834, rt:4161, total:60783) [2025-01-08 00:51:10.534445] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=114, ops=4, iops=118, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:10.534480] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:457, avg_bw:1868686, [delay/us]: prepare:5, schedule:2407, submit:2, device:736, total:3152) [2025-01-08 00:51:10.534497] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:74906, [delay/us]: prepare:16, schedule:770, submit:55834, device:4161, total:60783) [2025-01-08 00:51:10.534507] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:10.534530] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:15770, [delay/us]:prepare:5, schedule:106518, submit:0, rt:1752, total:108276) [2025-01-08 00:51:10.534553] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:35, bw:153762, [delay/us]:prepare:5, schedule:25514, submit:9460, rt:24288, total:59269) [2025-01-08 00:51:10.534562] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:10.534590] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:15771, [delay/us]: prepare:5, schedule:106518, submit:0, device:1752, total:108276) [2025-01-08 00:51:10.534605] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:35, avg_bw:153772, [delay/us]: prepare:5, schedule:25514, submit:9460, device:24288, total:59269) [2025-01-08 00:51:10.534615] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:10.534634] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3943, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:418, total:431) [2025-01-08 00:51:10.534658] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:10.534685] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3944, [delay/us]: prepare:0, schedule:0, submit:600000000, device:418, total:431) [2025-01-08 00:51:10.534703] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:10.534718] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:10.534726] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:10.534734] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:10.534741] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=715kB/s, net_out=331kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:10.534443] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46493][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:115179}, to_seq_no={branch:0, seq:7920}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44838} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.535977] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=15, stat_interval=5000000, avg (count/cost)=155, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:10.536004] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=26] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=15, stat_interval=5000000, avg (count/cost)=122353, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:10.536013] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=15, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:10.536021] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=15, stat_interval=5000000, avg (count/cost)=-2731, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:10.536029] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=15, stat_interval=5000000, avg (count/cost)=119776, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:10.539630] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D03B-0-0] [lt=42][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:10.541137] INFO [STORAGE.TRANS] rollback_to_savepoint (ob_trans_part_ctx.cpp:8738) [46389][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=40] rollback succeed trivially(trans_id={txid:44838}, ls_id={id:1}, op_sn=3, to_scn={branch:0, seq:7920}, last_scn={branch:0, seq:7920}) [2025-01-08 00:51:10.544352] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C628-0-0] [lt=48][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193775458183, cache_obj->added_lc()=false, cache_obj->get_object_id()=13606, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:10.544668] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2484) [46144][pnio1][T1][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=17][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:44838}, sender_ls_id={id:1}, receiver_ls_id={id:9223372036854775807}, msg_type=67, status=-4018, receiver_addr="192.168.159.139:2882", request_id=1736322504776635) [2025-01-08 00:51:10.544703] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:288) [46144][pnio1][T0][YB42C0A89F8B-00062B2D174BB864-0-0] [lt=32][errcode=0] trx rpc callback(ret=0, status=-4018, dst="192.168.159.139:2882", result={status:-4018, send_timestamp:1736326270541199, private_data:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:10.545854] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.546061] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.546295] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=20] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:14996353619, tenant_id:1}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:10.546329] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:10.546342] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8881fc3b0, thread=0x2ad90f1d3a70) [2025-01-08 00:51:10.546351] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:14996353619, tenant_id:1}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:10.546362] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7757417957, tenant_id:1001}, cur_thread_count=8, cnt=-1, new_thread_count=7) [2025-01-08 00:51:10.546373] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:10.546380] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6] try_wait return eagain(this=0x2ad8f90b0e70, thread=0x2ad91eec5e60) [2025-01-08 00:51:10.546385] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=5] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7757417957, tenant_id:1001}, cur_thread_count=8, cnt=-1, new_thread_count=7) [2025-01-08 00:51:10.552039] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46389][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=53][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963660780 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55698} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:8524} write_epoch=1, prev=this=0x2ad963654108 trans_version={val:1736326208260419000, v:0} scn={val:1736326208260419000, v:0} tx_id={txid:55597} prev=0x2ad963653f50 next=(nil) modify_count=118 acc_checksum=3751040381 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:47521} write_epoch=3, snapshot_version={val:1736326270546473000, v:0}, *this={this=0x2ad8c1457a98 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad963654108 latest_compact_node=0x2ad8dae86810 max_trans_version={val:1736326208260419000, v:0} max_trans_id=55597 max_elr_trans_version={val:1736326208260419000, v:0} max_elr_trans_id=55597 latest_compact_ts=1736326087180004 last_compact_cnt=113 total_trans_node_cnt=119 max_modify_scn={val:1736326208260419000, v:0} min_modify_scn={val:1736322641129989001, v:0}}) [2025-01-08 00:51:10.552102] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46389][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=61][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963660780 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55698} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:8524} write_epoch=1, ctx={this:0x2ad8ad855678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300444931, tablet_id:{id:309}, table_iter:0x2ad8ad853b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300444931, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270546473000, v:0}, tx_id:{txid:55698}, scn:{branch:0, seq:8521}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55698}, tx_desc:{this:0x2ad8b34ac8c0, tx_id:{txid:55698}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270548894, commit_ts:-1, finish_ts:-1, timeout_us:29999976, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270634311, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0434d0, mem_ctx:0x2ad89c044a50, tx_scn:{branch:0, seq:8524}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.552357] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46389][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=253][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55698}, data:0x2ad8ad852508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:8524}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.552444] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46389][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55698} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.552482] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46389][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=35][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.552493] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46389][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad91c00c6a0, local_datums:0x2ad91c00c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad91c00c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad91c00c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad91c00c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad91c00c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad91c00c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad91c00c373, hex: 484993F62D2B0600, int: 1736326270634312}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad91c00c37b, hex: 484993F62D2B0600, int: 1736326270634312}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad91c00c383, hex: C1BE050900000000, int: 151371457}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad91c00c38b, hex: C1BE050900000000, int: 151371457}]}) [2025-01-08 00:51:10.552695] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46389][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=159][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.552812] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46389][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB865-0-0] [lt=13] get number(ret=0, service_type_=0, range=1, base_id=1736326270548894000, start_id=1736326270548894000, end_id=1736326270548894001) [2025-01-08 00:51:10.563370] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=91] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:18076}, to_seq_no={branch:0, seq:8522}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55698} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.565432] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C95-0-0] [lt=128][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:10.567063] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C95-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:10.567798] INFO [STORAGE.TRANS] rollback_to_savepoint (ob_trans_part_ctx.cpp:8738) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=62] rollback succeed trivially(trans_id={txid:55698}, ls_id={id:1}, op_sn=3, to_scn={branch:0, seq:8522}, last_scn={branch:0, seq:8522}) [2025-01-08 00:51:10.570368] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:10.572689] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58410][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=65] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:51:10.579052] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2484) [46141][pnio1][T1][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=40][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:55698}, sender_ls_id={id:1}, receiver_ls_id={id:9223372036854775807}, msg_type=67, status=-4018, receiver_addr="192.168.159.138:2882", request_id=1736322504606915) [2025-01-08 00:51:10.579148] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:288) [46141][pnio1][T0][YB42C0A89F8A-00062B2D17C9C63A-0-0] [lt=93][errcode=0] trx rpc callback(ret=0, status=-4018, dst="192.168.159.138:2882", result={status:-4018, send_timestamp:1736326270567839, private_data:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:10.583085] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46502][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB866-0-0] [lt=63][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963660950 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44839} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:15953} write_epoch=1, prev=this=0x2ad8c1566038 trans_version={val:1736324387086570000, v:0} scn={val:1736324387086570000, v:0} tx_id={txid:42223} prev=(nil) next=(nil) modify_count=0 acc_checksum=3412027634 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:3556} write_epoch=1, snapshot_version={val:1736326270578918000, v:0}, *this={this=0x2ad8c1565fa8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1566038 latest_compact_node=(nil) max_trans_version={val:1736326209528605000, v:0} max_trans_id=44756 max_elr_trans_version={val:1736326209528605000, v:0} max_elr_trans_id=44756 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324387086570000, v:0} min_modify_scn={val:1736324387086570000, v:0}}) [2025-01-08 00:51:10.583154] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46502][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB866-0-0] [lt=67][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963660950 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44839} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:15953} write_epoch=1, ctx={this:0x2ad8b7dcf678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300473033, tablet_id:{id:309}, table_iter:0x2ad8b7dcdb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300473033, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270578918000, v:0}, tx_id:{txid:44839}, scn:{branch:0, seq:1227}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44839}, tx_desc:{this:0x2ad8b5fd5fe0, tx_id:{txid:44839}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270582361, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270658427, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:15953}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.583329] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46502][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB866-0-0] [lt=173][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44839}, data:0x2ad8b7dcc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:15953}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.583351] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46502][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB866-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44839} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.583384] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46502][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB866-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.583396] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46502][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB866-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad94ec0c6a0, local_datums:0x2ad94ec0c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad94ec0c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad94ec0c363, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c374, hex: F66492F62D2B0600, int: 1736326270575862}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c37c, hex: F66492F62D2B0600, int: 1736326270575862}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c384, hex: 00D0F70000000000, int: 16240640}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c38c, hex: 00D0F70000000000, int: 16240640}]}) [2025-01-08 00:51:10.583578] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46502][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB866-0-0] [lt=128][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.586665] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46384][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB866-0-0] [lt=27] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:18335}, to_seq_no={branch:0, seq:1228}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44839} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.601540] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3801-0-0] [lt=33] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:10.601606] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58384][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3801-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=13, avg_log_batch_cnt=0, total_group_log_size=1576, avg_group_log_size=121) [2025-01-08 00:51:10.615575] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.620178] INFO [LIB] stat (utility.h:1210) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A455F-0-0] [lt=19] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:10.620286] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58384][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A455F-0-0] [lt=96] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1333, avg_group_log_size=121) [2025-01-08 00:51:10.620345] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AC4D-0-0] [lt=16] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.839697837829589844e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=391, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.433372304119196770e+02) [2025-01-08 00:51:10.622301] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C63C-0-0] [lt=12][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963660bd0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55699} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:10453} write_epoch=1, prev=this=0x2ad8da45d4a0 trans_version={val:1736324386340473000, v:0} scn={val:1736324386340473000, v:0} tx_id={txid:52599} prev=(nil) next=(nil) modify_count=0 acc_checksum=3388464126 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:5965} write_epoch=1, snapshot_version={val:1736326270614718000, v:0}, *this={this=0x2ad8da45d410 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8da45d4a0 latest_compact_node=(nil) max_trans_version={val:1736326208302959000, v:0} max_trans_id=55598 max_elr_trans_version={val:1736326208302959000, v:0} max_elr_trans_id=55598 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386340473000, v:0} min_modify_scn={val:1736324386340473000, v:0}}) [2025-01-08 00:51:10.622370] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C63C-0-0] [lt=68][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963660bd0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55699} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:10453} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300514473, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300514473, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270614718000, v:0}, tx_id:{txid:55699}, scn:{branch:0, seq:7094}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55699}, tx_desc:{this:0x2ad8b082d120, tx_id:{txid:55699}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270621652, commit_ts:-1, finish_ts:-1, timeout_us:29999978, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270701389, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:10453}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.622463] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C63C-0-0] [lt=91][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55699}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:10453}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.622480] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C63C-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55699} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.622510] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C63C-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.622522] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C63C-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 4E4F94F62D2B0600, int: 1736326270701390}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 4E4F94F62D2B0600, int: 1736326270701390}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 710B080000000000, int: 527217}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00D03D0000000000, int: 4050944}]}) [2025-01-08 00:51:10.622610] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C63C-0-0] [lt=54][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.628535] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB868-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963660da0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44840} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:3412} write_epoch=1, prev=this=0x2ad8dae42eb0 trans_version={val:1736324509014921000, v:0} scn={val:1736324509014921000, v:0} tx_id={txid:42427} prev=0x2ad8c1575460 next=(nil) modify_count=4 acc_checksum=3656646905 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:1468} write_epoch=3, snapshot_version={val:1736326270626307000, v:0}, *this={this=0x2ad8c1566418 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8dae42eb0 latest_compact_node=(nil) max_trans_version={val:1736326209561059000, v:0} max_trans_id=44757 max_elr_trans_version={val:1736326209561059000, v:0} max_elr_trans_id=44757 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324509014921000, v:0} min_modify_scn={val:1736324387129570000, v:0}}) [2025-01-08 00:51:10.628622] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB868-0-0] [lt=85][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963660da0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44840} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:3412} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300525834, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300525834, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270626307000, v:0}, tx_id:{txid:44840}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44840}, tx_desc:{this:0x2ad8b082f180, tx_id:{txid:44840}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270627385, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270716457, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:3412}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.628723] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB868-0-0] [lt=99][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44840}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:3412}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.628740] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB868-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44840} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.628771] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB868-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.628783] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB868-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c374, hex: 6A3293F62D2B0600, int: 1736326270628458}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37c, hex: 6A3293F62D2B0600, int: 1736326270628458}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c384, hex: 00E0B60000000000, int: 11984896}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38c, hex: 00E0B60000000000, int: 11984896}]}) [2025-01-08 00:51:10.628882] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB868-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.629316] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63C-0-0] [lt=38] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:18481}, to_seq_no={branch:0, seq:7095}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55699} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.631134] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46386][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB868-0-0] [lt=46] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:3415}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44840} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.641779] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46386][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63D-0-0] [lt=344] get number(ret=0, service_type_=0, range=1, base_id=1736326270641542000, start_id=1736326270641542000, end_id=1736326270641542001) [2025-01-08 00:51:10.653171] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63E-0-0] [lt=13][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad9636610d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55700} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3497} write_epoch=1, prev=this=0x2ad8c1570bf0 trans_version={val:1736324508533827000, v:0} scn={val:1736324508533827000, v:0} tx_id={txid:52830} prev=0x2ad8c1570a38 next=(nil) modify_count=4 acc_checksum=2505866000 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:26113} write_epoch=3, snapshot_version={val:1736326270649787000, v:0}, *this={this=0x2ad8c1564d38 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8c1570bf0 latest_compact_node=(nil) max_trans_version={val:1736326208336961000, v:0} max_trans_id=55599 max_elr_trans_version={val:1736326208336961000, v:0} max_elr_trans_id=55599 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324508533827000, v:0} min_modify_scn={val:1736324386362509000, v:0}}) [2025-01-08 00:51:10.653323] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63E-0-0] [lt=149][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad9636610d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55700} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3497} write_epoch=1, ctx={this:0x2ad8b7dcf678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300550126, tablet_id:{id:309}, table_iter:0x2ad8b7dcdb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300550126, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270649787000, v:0}, tx_id:{txid:55700}, scn:{branch:0, seq:3494}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55700}, tx_desc:{this:0x2ad8b5fd8040, tx_id:{txid:55700}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270652088, commit_ts:-1, finish_ts:-1, timeout_us:29999979, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270739793, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:3497}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.653433] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63E-0-0] [lt=108][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55700}, data:0x2ad8b7dcc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:3497}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.653451] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63E-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55700} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.653484] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63E-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.653497] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63E-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad94ec0c6a0, local_datums:0x2ad94ec0c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad94ec0c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad94ec0c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c373, hex: 52E594F62D2B0600, int: 1736326270739794}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c37b, hex: 52E594F62D2B0600, int: 1736326270739794}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c383, hex: 86DA080000000000, int: 580230}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c38b, hex: 00802F0000000000, int: 3112960}]}) [2025-01-08 00:51:10.653617] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63E-0-0] [lt=66][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.657305] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46498][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63E-0-0] [lt=39] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7643}, to_seq_no={branch:0, seq:3495}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55700} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.657455] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB86B-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326270656235000, start_id=1736326270656235000, end_id=1736326270656235001) [2025-01-08 00:51:10.670140] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46497][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C63F-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326270669555000, start_id=1736326270669555000, end_id=1736326270669555001) [2025-01-08 00:51:10.671201] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86C-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963661520 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44842} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:8417} write_epoch=1, prev=this=0x2ad8c14586c8 trans_version={val:1736322641178398000, v:0} scn={val:1736322641178398000, v:0} tx_id={txid:40115} prev=(nil) next=(nil) modify_count=0 acc_checksum=1913904902 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:2356} write_epoch=1, snapshot_version={val:1736326270668382000, v:0}, *this={this=0x2ad8c1458638 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c14586c8 latest_compact_node=(nil) max_trans_version={val:1736326209650751000, v:0} max_trans_id=44759 max_elr_trans_version={val:1736326209650751000, v:0} max_elr_trans_id=44759 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641178398000, v:0} min_modify_scn={val:1736322641178398000, v:0}}) [2025-01-08 00:51:10.671323] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86C-0-0] [lt=120][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963661520 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44842} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:8417} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300568736, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300568736, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270668382000, v:0}, tx_id:{txid:44842}, scn:{branch:0, seq:4945}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44842}, tx_desc:{this:0x2ad8b083b3c0, tx_id:{txid:44842}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270670826, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270753540, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:8417}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.671479] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86C-0-0] [lt=154][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44842}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:8417}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.671500] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86C-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44842} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.671532] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86C-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.671543] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86C-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 99D793F62D2B0600, int: 1736326270670745}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 99D793F62D2B0600, int: 1736326270670745}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 00C0CF0000000000, int: 13615104}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00C0CF0000000000, int: 13615104}]}) [2025-01-08 00:51:10.671674] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86C-0-0] [lt=81][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.677506] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB86C-0-0] [lt=288] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:15474}, to_seq_no={branch:0, seq:4946}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44842} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.686713] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C640-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad9636616f8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55701} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:6304} write_epoch=1, prev=this=0x2ad8c1565188 trans_version={val:1736324386386506000, v:0} scn={val:1736324386386506000, v:0} tx_id={txid:52601} prev=(nil) next=(nil) modify_count=0 acc_checksum=370822477 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:9654} write_epoch=1, snapshot_version={val:1736326270680701000, v:0}, *this={this=0x2ad8c15650f8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1565188 latest_compact_node=(nil) max_trans_version={val:1736326208382151000, v:0} max_trans_id=55600 max_elr_trans_version={val:1736326208382151000, v:0} max_elr_trans_id=55600 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386386506000, v:0} min_modify_scn={val:1736324386386506000, v:0}}) [2025-01-08 00:51:10.686843] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C640-0-0] [lt=128][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad9636616f8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55701} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:6304} write_epoch=1, ctx={this:0x2ad8b7dcf678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300582008, tablet_id:{id:309}, table_iter:0x2ad8b7dcdb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300582008, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270680701000, v:0}, tx_id:{txid:55701}, scn:{branch:0, seq:6301}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55701}, tx_desc:{this:0x2ad8b5fda0a0, tx_id:{txid:55701}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270685673, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270771275, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:6304}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.687012] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C640-0-0] [lt=166][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55701}, data:0x2ad8b7dcc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:6304}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.687034] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C640-0-0] [lt=21][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55701} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.687116] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C640-0-0] [lt=79][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.687129] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C640-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad94ec0c6a0, local_datums:0x2ad94ec0c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad94ec0c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad94ec0c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c375, hex: 336A95F62D2B0600, int: 1736326270773811}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c37d, hex: 336A95F62D2B0600, int: 1736326270773811}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad94ec0c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.687310] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C640-0-0] [lt=79][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.688504] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C640-0-0] [lt=52] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8938}, to_seq_no={branch:0, seq:6302}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55701} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.689346] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:10.692803] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58384][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=15] worker thread exit(tenant_->id()=1002, workers_.get_size()=5) [2025-01-08 00:51:10.706960] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86E-0-0] [lt=13][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963661978 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44843} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:5487} write_epoch=1, prev=this=0x2ad8dae40ec0 trans_version={val:1736324387223787000, v:0} scn={val:1736324387223787000, v:0} tx_id={txid:42227} prev=(nil) next=(nil) modify_count=0 acc_checksum=500068692 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7190} write_epoch=1, snapshot_version={val:1736326270703098000, v:0}, *this={this=0x2ad8dae40e30 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae40ec0 latest_compact_node=(nil) max_trans_version={val:1736326209678317000, v:0} max_trans_id=44760 max_elr_trans_version={val:1736326209678317000, v:0} max_elr_trans_id=44760 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324387223787000, v:0} min_modify_scn={val:1736324387223787000, v:0}}) [2025-01-08 00:51:10.707068] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86E-0-0] [lt=106][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963661978 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44843} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:5487} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300603645, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300603645, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270703098000, v:0}, tx_id:{txid:44843}, scn:{branch:0, seq:1429}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44843}, tx_desc:{this:0x2ad8b0843540, tx_id:{txid:44843}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270706663, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270793177, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:5487}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.707169] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86E-0-0] [lt=100][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44843}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:5487}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.707317] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86E-0-0] [lt=145][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44843} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.707358] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86E-0-0] [lt=38][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.707371] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86E-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: B95794F62D2B0600, int: 1736326270703545}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: B95794F62D2B0600, int: 1736326270703545}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 00E0010000000000, int: 122880}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00E0010000000000, int: 122880}]}) [2025-01-08 00:51:10.707510] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB86E-0-0] [lt=73][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.707851] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:2312K/566:999212 disk:2312K/566:999212 t1m2:160K/34:326344 t1m2g0:160K/34:326344 t1002m2:176K/41:8380289 t1002m2g0:176K/41:8380289 t1001m2:1976K/491:427732 t1001m2g0:1976K/491:427732 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:10.710549] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB86E-0-0] [lt=40] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7975}, to_seq_no={branch:0, seq:1430}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44843} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.713060] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=91, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:10.713267] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:10.714942] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58435][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:10.715901] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.716032] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5328, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:10.716055] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1699) [2025-01-08 00:51:10.716075] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=555, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:10.716152] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=146] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:10.717870] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:51:10.720461] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8521-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:10.720498] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8521-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=33388, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:10.720512] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8521-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=21784, avg_log_gen_to_submit_time=21907, avg_log_submit_to_flush_time=5944, avg_log_submit_to_slide_time=11480) [2025-01-08 00:51:10.720591] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1699, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:10.722996] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46390][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB86F-0-0] [lt=13] get number(ret=0, service_type_=0, range=1, base_id=1736326270722665000, start_id=1736326270722665000, end_id=1736326270722665001) [2025-01-08 00:51:10.723158] INFO [LIB] stat (utility.h:1210) [46390][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB86F-0-0] [lt=30] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=34, stat_interval=1000000, avg (count/cost)=76, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:10.723263] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46390][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB86F-0-0] [lt=92] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=34, avg_log_batch_cnt=1, total_group_log_size=21785, avg_group_log_size=640) [2025-01-08 00:51:10.723728] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:10.725052] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:10.725387] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] [CACHE] tenant_id= 1 cache memory info: [CACHE] tenant_id= 1 | cache_name= schema_cache | cache_size= 54098944 | cache_store_size= 52019200 | cache_map_size= 2079744 | kv_cnt= 3298 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tablet_table_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 10 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= vtable_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 54 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1176 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_block_cache | cache_size= 20806656 | cache_store_size= 18726912 | cache_map_size= 2079744 | kv_cnt= 2397 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 22 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= bf_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 3 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= fuse_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 56 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= storage_meta_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 1126 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 8 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= log_kv_cache | cache_size= 89472000 | cache_store_size= 87392256 | cache_map_size= 2079744 | kv_cnt= 1285 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 191 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 275 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_ds_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 31 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-01-08 00:51:10.725535] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=15] [CACHE] tenant_id= 1001 cache memory info: [CACHE] tenant_id= 1001 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 989 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1514 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= storage_meta_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 992 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= log_kv_cache | cache_size= 47856640 | cache_store_size= 45776896 | cache_map_size= 2079744 | kv_cnt= 658 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 58 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 242 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-01-08 00:51:10.725598] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=13] [CACHE] tenant_id= 1002 cache memory info: [CACHE] tenant_id= 1002 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 710 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1066 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 712 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= log_kv_cache | cache_size= 24968192 | cache_store_size= 22888448 | cache_map_size= 2079744 | kv_cnt= 317 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 47 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 234 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-01-08 00:51:10.725618] INFO print_current_status (ob_kvcache_hazard_version.cpp:328) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=9] [KVCACHE-HAZARD] hazard version status info: current_version: 11173 | min_version= 11173 | total_nodes_count: 0 | [2025-01-08 00:51:10.725917] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=62] [PALF STAT WRITE LOG COST TIME](cur_stat_count=35, stat_interval=1000000, avg (count/cost)=3797, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:10.725941] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=23368) [2025-01-08 00:51:10.725961] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=35, stat_interval=1000000, avg (count/cost)=1326, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:10.725987] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=35, stat_interval=1000000, avg (count/cost)=46, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:10.727862] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B7E-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=35, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:10.727886] INFO [LIB] stat (utility.h:1210) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B7E-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=35, stat_interval=1000000, avg (count/cost)=37472, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:10.727897] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B7E-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=35, avg_log_gen_to_freeze_time=28045, avg_log_gen_to_submit_time=28143, avg_log_submit_to_flush_time=5517, avg_log_submit_to_slide_time=9328) [2025-01-08 00:51:10.728002] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C642-0-0] [lt=20][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8ad00 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55702} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:10817} write_epoch=1, prev=this=0x2ad8c1565570 trans_version={val:1736324386414645000, v:0} scn={val:1736324386414645000, v:0} tx_id={txid:52602} prev=(nil) next=(nil) modify_count=0 acc_checksum=952487989 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:18441} write_epoch=1, snapshot_version={val:1736326270722665001, v:0}, *this={this=0x2ad8c15654e0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1565570 latest_compact_node=(nil) max_trans_version={val:1736326208450263000, v:0} max_trans_id=55601 max_elr_trans_version={val:1736326208450263000, v:0} max_elr_trans_id=55601 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386414645000, v:0} min_modify_scn={val:1736324386414645000, v:0}}) [2025-01-08 00:51:10.728060] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C642-0-0] [lt=56][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8ad00 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55702} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:10817} write_epoch=1, ctx={this:0x2ad8b7ed3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300620612, tablet_id:{id:309}, table_iter:0x2ad8b7ed1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300620612, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270722665001, v:0}, tx_id:{txid:55702}, scn:{branch:0, seq:8518}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55702}, tx_desc:{this:0x2ad8b34bab60, tx_id:{txid:55702}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270726887, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270808429, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0434d0, mem_ctx:0x2ad89c044a50, tx_scn:{branch:0, seq:10817}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.728278] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C642-0-0] [lt=215][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55702}, data:0x2ad8b7ed0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:10817}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.728282] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=56, total_batch_msg_count=0) [2025-01-08 00:51:10.728296] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C642-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55702} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.728329] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C642-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.728333] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=23368, hit_cnt=35, read_cnt=35, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:10.728342] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C642-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad95a20c6a0, local_datums:0x2ad95a20c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad95a20c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad95a20c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c375, hex: 6EF195F62D2B0600, int: 1736326270808430}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c37d, hex: 6EF195F62D2B0600, int: 1736326270808430}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.728549] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C642-0-0] [lt=73][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.729684] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=27] ====== tenant freeze timer task ====== [2025-01-08 00:51:10.730521] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C642-0-0] [lt=42] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:13265}, to_seq_no={branch:0, seq:8519}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55702} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.737391] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB870-0-0] [lt=18][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963661bf8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44844} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4795} write_epoch=1, prev=this=0x2ad963659240 trans_version={val:1736326209710957000, v:0} scn={val:1736326209710957000, v:0} tx_id={txid:44761} prev=0x2ad8dae89d40 next=(nil) modify_count=60 acc_checksum=2209893249 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:19324} write_epoch=3, snapshot_version={val:1736326270734416000, v:0}, *this={this=0x2ad8da45dba8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad963659240 latest_compact_node=0x2ad8dae89d40 max_trans_version={val:1736326209710957000, v:0} max_trans_id=44761 max_elr_trans_version={val:1736326209710957000, v:0} max_elr_trans_id=44761 latest_compact_ts=1736326209717184 last_compact_cnt=59 total_trans_node_cnt=61 max_modify_scn={val:1736326209710957000, v:0} min_modify_scn={val:1736324387226061003, v:0}}) [2025-01-08 00:51:10.737454] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB870-0-0] [lt=61][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963661bf8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44844} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4795} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300633224, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300633224, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270734416000, v:0}, tx_id:{txid:44844}, scn:{branch:0, seq:4792}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44844}, tx_desc:{this:0x2ad8b08455a0, tx_id:{txid:44844}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270736848, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270821692, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:4795}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.737549] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB870-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44844}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4795}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.737567] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB870-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44844} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.737599] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB870-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.737612] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB870-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: AED494F62D2B0600, int: 1736326270735534}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: AED494F62D2B0600, int: 1736326270735534}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 00D0240000000000, int: 2412544}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00D0240000000000, int: 2412544}]}) [2025-01-08 00:51:10.737760] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB870-0-0] [lt=71][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.739415] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB870-0-0] [lt=177] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8305}, to_seq_no={branch:0, seq:4793}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44844} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.739834] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46499][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C643-0-0] [lt=107] get number(ret=0, service_type_=0, range=1, base_id=1736326270739297000, start_id=1736326270739297000, end_id=1736326270739297001) [2025-01-08 00:51:10.747766] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D3A-0-0] [lt=15] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:10.749637] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5221648707, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:10.749666] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:10.749682] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8cf1fba60) [2025-01-08 00:51:10.749696] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5221648707, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:10.754406] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58436][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:10.754497] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:51:10.760688] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE83D-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326270760576000, start_id=1736326270760576000, end_id=1736326270760576001) [2025-01-08 00:51:10.762534] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE83E-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1736326270761753000, start_id=1736326270761753000, end_id=1736326270761753001) [2025-01-08 00:51:10.765970] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58436][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6819-0-0] [lt=40] init point thread id with(&point=0x5572d50e3a40, idx_=1092, point=[thread id=58436, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58436) [2025-01-08 00:51:10.767019] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=39] PNIO [ratelimit] time: 1736326270767016, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006368, add_bytes: 0 [2025-01-08 00:51:10.767715] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58437][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:10.767857] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:51:10.773388] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C646-0-0] [lt=106][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8aed0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55704} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:2322} write_epoch=1, prev=this=0x2ad963655098 trans_version={val:1736326208533777000, v:0} scn={val:1736326208533777000, v:0} tx_id={txid:55603} prev=0x2ad8dae889c8 next=(nil) modify_count=60 acc_checksum=364466506 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:22760} write_epoch=3, snapshot_version={val:1736326270769557001, v:0}, *this={this=0x2ad8c1565b38 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad963655098 latest_compact_node=0x2ad8dae889c8 max_trans_version={val:1736326208533777000, v:0} max_trans_id=55603 max_elr_trans_version={val:1736326208533777000, v:0} max_elr_trans_id=55603 latest_compact_ts=1736326208540967 last_compact_cnt=59 total_trans_node_cnt=61 max_modify_scn={val:1736326208533777000, v:0} min_modify_scn={val:1736324386665957000, v:0}}) [2025-01-08 00:51:10.773572] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C646-0-0] [lt=182][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8aed0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55704} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:2322} write_epoch=1, ctx={this:0x2ad8b7ed3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300671390, tablet_id:{id:309}, table_iter:0x2ad8b7ed1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300671390, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270769557001, v:0}, tx_id:{txid:55704}, scn:{branch:0, seq:2319}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55704}, tx_desc:{this:0x2ad8b34c4d40, tx_id:{txid:55704}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270769557, commit_ts:-1, finish_ts:-1, timeout_us:29999915, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270861671, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0434d0, mem_ctx:0x2ad89c044a50, tx_scn:{branch:0, seq:2322}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.773697] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C646-0-0] [lt=123][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55704}, data:0x2ad8b7ed0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:2322}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.773774] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C646-0-0] [lt=75][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55704} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.773806] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C646-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.773961] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C646-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad95a20c6a0, local_datums:0x2ad95a20c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad95a20c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad95a20c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c373, hex: 76CA96F62D2B0600, int: 1736326270863990}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c37b, hex: 76CA96F62D2B0600, int: 1736326270863990}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c383, hex: 9DBF920600000000, int: 110280605}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad95a20c38b, hex: 9DBF920600000000, int: 110280605}]}) [2025-01-08 00:51:10.774057] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46504][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C646-0-0] [lt=63][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.776403] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C646-0-0] [lt=40] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5780}, to_seq_no={branch:0, seq:2320}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55704} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.776870] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58435][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA681A-0-0] [lt=70] init point thread id with(&point=0x5572d50e3a00, idx_=1091, point=[thread id=58435, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58435) [2025-01-08 00:51:10.777497] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326270522134001, v:0}, new_max_version={val:1736326270522134001, v:0}, last_min_version={val:1736326270185992000, v:0}, last_max_version={val:1736326270185992000, v:0}, total_time_us=7837, delta_us=255360) [2025-01-08 00:51:10.777527] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326270522134001, v:0}, min_version={val:1736326270522134001, v:0}, max_version={val:1736326270522134001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:10.777547] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:10.782507] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:10.782657] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=39] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:10.789519] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB874-0-0] [lt=65][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8da4c3908 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44846} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7904} write_epoch=1, prev=this=0x2ad8c1566f48 trans_version={val:1736324387335228000, v:0} scn={val:1736324387335228000, v:0} tx_id={txid:42230} prev=(nil) next=(nil) modify_count=0 acc_checksum=1788252403 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:19273} write_epoch=1, snapshot_version={val:1736326270785586000, v:0}, *this={this=0x2ad8c1566eb8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1566f48 latest_compact_node=(nil) max_trans_version={val:1736326209757446000, v:0} max_trans_id=44763 max_elr_trans_version={val:1736326209757446000, v:0} max_elr_trans_id=44763 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324387335228000, v:0} min_modify_scn={val:1736324387335228000, v:0}}) [2025-01-08 00:51:10.789597] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB874-0-0] [lt=77][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8da4c3908 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44846} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7904} write_epoch=1, ctx={this:0x2ad8b7855678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300686070, tablet_id:{id:309}, table_iter:0x2ad8b7853b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300686070, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270785586000, v:0}, tx_id:{txid:44846}, scn:{branch:0, seq:4449}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44846}, tx_desc:{this:0x2ad8b95588a0, tx_id:{txid:44846}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270789313, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270871140, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0191d0, mem_ctx:0x2ad89c01a750, tx_scn:{branch:0, seq:7904}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.789693] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB874-0-0] [lt=93][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44846}, data:0x2ad8b7852508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:7904}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.789710] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB874-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44846} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.789742] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB874-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.789753] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB874-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96a00c6a0, local_datums:0x2ad96a00c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96a00c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96a00c363, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c373, hex: 429F95F62D2B0600, int: 1736326270787394}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c37b, hex: 429F95F62D2B0600, int: 1736326270787394}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c383, hex: 0000BA0000000000, int: 12189696}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96a00c38b, hex: 0000BA0000000000, int: 12189696}]}) [2025-01-08 00:51:10.789852] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB874-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.790716] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:10.791752] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB874-0-0] [lt=100] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9351}, to_seq_no={branch:0, seq:4450}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44846} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.793807] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F5-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:10.793904] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F5-0-0] [lt=76] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326270256872005, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326270256872005, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:10.800912] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8A-00062B2D17C9C648-0-0] [lt=40] get number(ret=0, service_type_=0, range=1, base_id=1736326270800378000, start_id=1736326270800378000, end_id=1736326270800378001) [2025-01-08 00:51:10.805624] INFO pn_ratelimit (group.c:653) [58411][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=23] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:10.806296] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:10.807548] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=8] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=647712768, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=426029056, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:10.807652] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=57] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:10.807693] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=38] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2821, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:10.807817] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=74] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:10.807895] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=45] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:10.807913] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=16] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:10.808451] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=4975, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:10.808493] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1820) [2025-01-08 00:51:10.808621] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=118] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=8241, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:10.808688] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=47, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:10.808846] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:10.809628] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C648-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963662728 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55705} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:9783} write_epoch=1, prev=this=0x2ad8da4c2328 trans_version={val:1736326208571630000, v:0} scn={val:1736326208571630000, v:0} tx_id={txid:55604} prev=0x2ad8dae88c50 next=(nil) modify_count=60 acc_checksum=1376677543 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:32691} write_epoch=3, snapshot_version={val:1736326270800378000, v:0}, *this={this=0x2ad8da45d7d0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8da4c2328 latest_compact_node=0x2ad8dae88c50 max_trans_version={val:1736326208571630000, v:0} max_trans_id=55604 max_elr_trans_version={val:1736326208571630000, v:0} max_elr_trans_id=55604 latest_compact_ts=1736326208586977 last_compact_cnt=59 total_trans_node_cnt=61 max_modify_scn={val:1736326208571630000, v:0} min_modify_scn={val:1736324386993883002, v:0}}) [2025-01-08 00:51:10.809819] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C648-0-0] [lt=188][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963662728 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55705} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:9783} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300706775, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300706775, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270800378000, v:0}, tx_id:{txid:55705}, scn:{branch:0, seq:3590}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55705}, tx_desc:{this:0x2ad8b085b9c0, tx_id:{txid:55705}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270809088, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270890253, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:9783}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.809929] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C648-0-0] [lt=108][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55705}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:9783}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.809949] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C648-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55705} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.809979] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C648-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.809989] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C648-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 7C3A97F62D2B0600, int: 1736326270892668}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 7C3A97F62D2B0600, int: 1736326270892668}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 40EA801100000000, int: 293661248}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 40EA801100000000, int: 293661248}]}) [2025-01-08 00:51:10.810151] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C648-0-0] [lt=122][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.811496] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:10.812423] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C648-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:13663}, to_seq_no={branch:0, seq:3591}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55705} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.813898] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58400][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=41] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:51:10.816063] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.816507] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58429][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=72] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:10.820262] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50612][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F5-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326270819408000, start_id=1736326270819408000, end_id=1736326270819408001) [2025-01-08 00:51:10.820393] INFO [LIB] stat (utility.h:1210) [50612][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F5-0-0] [lt=30] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=83, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:10.820419] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50612][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F5-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=10804, avg_group_log_size=600) [2025-01-08 00:51:10.820436] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50612][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F5-0-0] [lt=13] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10054) [2025-01-08 00:51:10.823322] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=36] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=5151, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:10.823348] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=11393) [2025-01-08 00:51:10.823789] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=432] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=2182, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:10.823846] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=87] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10643) [2025-01-08 00:51:10.823909] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=41, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:10.825320] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB876-0-0] [lt=35][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8b158 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44847} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:10794} write_epoch=1, prev=this=0x2ad8c156f8a8 trans_version={val:1736324448500937000, v:0} scn={val:1736324448500937000, v:0} tx_id={txid:42342} prev=0x2ad8c156f6f0 next=(nil) modify_count=2 acc_checksum=3147207185 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:40876} write_epoch=3, snapshot_version={val:1736326270816481000, v:0}, *this={this=0x2ad8c1567298 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad8c156f8a8 latest_compact_node=(nil) max_trans_version={val:1736326209796553000, v:0} max_trans_id=44764 max_elr_trans_version={val:1736326209796553000, v:0} max_elr_trans_id=44764 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=3 max_modify_scn={val:1736324448500937000, v:0} min_modify_scn={val:1736324387344938000, v:0}}) [2025-01-08 00:51:10.825472] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB876-0-0] [lt=149][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8b158 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44847} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:10794} write_epoch=1, ctx={this:0x2ad8b7d51678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300718789, tablet_id:{id:309}, table_iter:0x2ad8b7d4fb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300718789, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270816481000, v:0}, tx_id:{txid:44847}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44847}, tx_desc:{this:0x2ad8b60a8800, tx_id:{txid:44847}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270823931, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270905074, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:10794}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.825612] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB876-0-0] [lt=90][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44847}, data:0x2ad8b7d4e508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:10794}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.825635] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB876-0-0] [lt=22][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44847} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.825666] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB876-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.825677] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB876-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad960c0c6a0, local_datums:0x2ad960c0c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad960c0c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad960c0c363, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c373, hex: DC1396F62D2B0600, int: 1736326270817244}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c37b, hex: DC1396F62D2B0600, int: 1736326270817244}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c383, hex: 0060870000000000, int: 8871936}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c38b, hex: 0060870000000000, int: 8871936}]}) [2025-01-08 00:51:10.825764] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA390D-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=16, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:10.825787] INFO [LIB] stat (utility.h:1210) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA390D-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=13681, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:10.825793] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB876-0-0] [lt=54][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.825797] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58385][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA390D-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=24686, avg_log_gen_to_submit_time=24781, avg_log_submit_to_flush_time=8490, avg_log_submit_to_slide_time=13230) [2025-01-08 00:51:10.826330] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=19] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:10.826373] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=11393, hit_cnt=19, read_cnt=19, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:10.829860] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB876-0-0] [lt=91] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:15831}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44847} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.848315] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46501][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64A-0-0] [lt=13][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8b328 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55706} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3678} write_epoch=1, prev=this=0x2ad8c146cb98 trans_version={val:1736322701405270000, v:0} scn={val:1736322701405270000, v:0} tx_id={txid:50200} prev=0x2ad8c146c9e0 next=(nil) modify_count=2 acc_checksum=953597501 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3524} write_epoch=3, snapshot_version={val:1736326270843042000, v:0}, *this={this=0x2ad8c1458bf8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad8c146cb98 latest_compact_node=(nil) max_trans_version={val:1736326208610792000, v:0} max_trans_id=55605 max_elr_trans_version={val:1736326208610792000, v:0} max_elr_trans_id=55605 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=3 max_modify_scn={val:1736322701405270000, v:0} min_modify_scn={val:1736322641212342001, v:0}}) [2025-01-08 00:51:10.848386] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46501][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64A-0-0] [lt=69][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8b328 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55706} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3678} write_epoch=1, ctx={this:0x2ad8b7d51678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300745740, tablet_id:{id:309}, table_iter:0x2ad8b7d4fb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300745740, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270843042000, v:0}, tx_id:{txid:55706}, scn:{branch:0, seq:3675}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55706}, tx_desc:{this:0x2ad8b60ac8c0, tx_id:{txid:55706}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270847731, commit_ts:-1, finish_ts:-1, timeout_us:29999988, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270932991, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:3678}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.848480] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46501][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64A-0-0] [lt=93][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55706}, data:0x2ad8b7d4e508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:3678}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.848497] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46501][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64A-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55706} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.848530] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46501][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64A-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.848543] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46501][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64A-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad960c0c6a0, local_datums:0x2ad960c0c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad960c0c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad960c0c363, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c374, hex: 16E197F62D2B0600, int: 1736326270935318}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c37c, hex: 16E197F62D2B0600, int: 1736326270935318}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c384, hex: 00D0420300000000, int: 54710272}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad960c0c38c, hex: 00D0420300000000, int: 54710272}]}) [2025-01-08 00:51:10.848646] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46501][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64A-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.850430] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=26] Slow write(duration=10940, write_len=4096) [2025-01-08 00:51:10.850460] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8629, offset=2448691}, end_cursor:ObLogCursor{file_id=1, log_id=8630, offset=2448996}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947c4050, len:717, log_data_len:305, seq:8629, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:10.851494] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4C-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:10.851543] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4C-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326270613134000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326270613134000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:10.852419] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64A-0-0] [lt=136] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10949}, to_seq_no={branch:0, seq:3676}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55706} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.859364] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46385][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB877-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326270858229000, start_id=1736326270858229000, end_id=1736326270858229001) [2025-01-08 00:51:10.861353] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E4C-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326270861294000, start_id=1736326270861294000, end_id=1736326270861294001) [2025-01-08 00:51:10.872090] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=18] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8630, offset=2448996}, end_cursor:ObLogCursor{file_id=1, log_id=8631, offset=2449293}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947c8050, len:412, log_data_len:297, seq:8630, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:10.873340] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4C-0-0] [lt=178] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:10.873403] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4C-0-0] [lt=42] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326270613134002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326270613134002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:10.888048] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDE1-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193775801880, cache_obj->added_lc()=false, cache_obj->get_object_id()=13607, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:10.891520] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB878-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963662b78 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44849} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:85} seq_no={branch:0, seq:10111} write_epoch=1, prev=this=0x2ad8c1458aa0 trans_version={val:1736322641212342000, v:0} scn={val:1736322641212342000, v:0} tx_id={txid:40117} prev=(nil) next=(nil) modify_count=0 acc_checksum=3184665992 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:85} seq_no={branch:0, seq:4058} write_epoch=1, snapshot_version={val:1736326270881072000, v:0}, *this={this=0x2ad8c1458a10 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1458aa0 latest_compact_node=(nil) max_trans_version={val:1736326209822010000, v:0} max_trans_id=44765 max_elr_trans_version={val:1736326209822010000, v:0} max_elr_trans_id=44765 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641212342000, v:0} min_modify_scn={val:1736322641212342000, v:0}}) [2025-01-08 00:51:10.891575] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB878-0-0] [lt=54][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963662b78 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44849} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:85} seq_no={branch:0, seq:10111} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300782407, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300782407, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270881072000, v:0}, tx_id:{txid:44849}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44849}, tx_desc:{this:0x2ad8b08820e0, tx_id:{txid:44849}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270889277, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270969100, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:10111}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.891678] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB878-0-0] [lt=99][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44849}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:10111}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:10.891725] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB878-0-0] [lt=45][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44849} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.891765] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB878-0-0] [lt=37][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:10.891778] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB878-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c372, hex: 651497F62D2B0600, int: 1736326270882917}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37a, hex: 651497F62D2B0600, int: 1736326270882917}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:10.891947] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D174BB878-0-0] [lt=112][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:10.892192] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C64C-0-0] [lt=21][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963662d40 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55709} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:10115} write_epoch=1, prev=this=0x2ad8c146cff0 trans_version={val:1736322701423200000, v:0} scn={val:1736322701423200000, v:0} tx_id={txid:50201} prev=0x2ad8c146ce38 next=(nil) modify_count=2 acc_checksum=1781982909 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:18} write_epoch=3, snapshot_version={val:1736326270886077000, v:0}, *this={this=0x2ad8c14590f0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad8c146cff0 latest_compact_node=(nil) max_trans_version={val:1736326208668840000, v:0} max_trans_id=55606 max_elr_trans_version={val:1736326208668840000, v:0} max_elr_trans_id=55606 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=3 max_modify_scn={val:1736322701423200000, v:0} min_modify_scn={val:1736322641259146000, v:0}}) [2025-01-08 00:51:10.892247] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C64C-0-0] [lt=54][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963662d40 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55709} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:10115} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300782279, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300782279, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326270886077000, v:0}, tx_id:{txid:55709}, scn:{branch:0, seq:10110}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55709}, tx_desc:{this:0x2ad8b0879f60, tx_id:{txid:55709}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326270889277, commit_ts:-1, finish_ts:-1, timeout_us:29999932, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326270969098, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:10115}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:10.892374] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C64C-0-0] [lt=126][errcode=-5024] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:10.892393] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C64C-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55709} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.894842] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB878-0-0] [lt=35] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:17529}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44849} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.895899] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64C-0-0] [lt=40] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:17532}, to_seq_no={branch:0, seq:10111}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55709} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.899779] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=81] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.899821] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.899844] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.899877] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:10.911252] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46383][T1_L0_G0][T1][YB42C0A89F8B-00062B2D174BB879-0-0] [lt=121] get number(ret=0, service_type_=0, range=1, base_id=1736326270908340000, start_id=1736326270908340000, end_id=1736326270908340001) [2025-01-08 00:51:10.914312] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18668290}, readable_log_tail:{lsn:18668290}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48436478, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:10.914413] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=99] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:10.914427] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2163658}, readable_log_tail:{lsn:2163658}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64941110, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:10.914478] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=50] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:10.914489] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] get_replay_process success(id={id:1002}, submitted_log_size=2196816, unsubmitted_log_size=2196816, replayed_log_size=2196816, unreplayed_log_size=0) [2025-01-08 00:51:10.914499] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] get_replay_process success(id={id:1003}, submitted_log_size=2184651, unsubmitted_log_size=2184651, replayed_log_size=2184651, unreplayed_log_size=0) [2025-01-08 00:51:10.914527] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:10.916500] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=102][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:10.918179] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=135] Slow write(duration=12878, write_len=8192) [2025-01-08 00:51:10.918535] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=351] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8631, offset=2449293}, end_cursor:ObLogCursor{file_id=1, log_id=8632, offset=2449590}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947cc050, len:4211, log_data_len:297, seq:8631, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:10.921239] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=19] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8632, offset=2449590}, end_cursor:ObLogCursor{file_id=1, log_id=8633, offset=2449887}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947d0050, len:3914, log_data_len:297, seq:8632, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:10.924043] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:10.924836] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58438][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:10.925133] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92] worker thread created(tenant_->id()=1002, group_id_=1, token=5) [2025-01-08 00:51:10.928628] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29F-0-0] [lt=176] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:10.928695] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C29F-0-0] [lt=46] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326270511849002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326270511849002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:10.929658] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=114] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:10.930655] INFO [COMMON] wash (ob_kvcache_store.cpp:356) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] Wash time detail, (compute_wash_size_time=257, refresh_score_time=338, wash_time=0) [2025-01-08 00:51:10.949788] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C64E-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55713} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:10.950102] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=27] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15070210559, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:10.950301] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=194][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:10.950313] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x5572d527f900, thread=0x2ad8bf5d5290) [2025-01-08 00:51:10.950323] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15070210559, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:10.950334] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7793968495, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:10.950345] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:10.950351] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6] try_wait return eagain(this=0x2ad8c0eb0e70, thread=0x2ad934da1e10) [2025-01-08 00:51:10.950357] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=5] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7793968495, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:10.955169] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C64E-0-0] [lt=37] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:20004}, to_seq_no={branch:0, seq:15438}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55713} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:10.974015] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326270769557002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=204445, in_cluster_service=true, cluster_version={val:1736326270522134001, v:0}, min_cluster_version={val:1736326270522134001, v:0}, max_cluster_version={val:1736326270522134001, v:0}, get_cluster_version_err=0, cluster_version_delta=451868, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36434, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=525470989, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:11.004726] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C650-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55714} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:11.015362] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46497][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C650-0-0] [lt=98] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:23814}, to_seq_no={branch:0, seq:5372}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55714} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:11.016768] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.017811] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [46106][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=69][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5024, dropped:16, tid:46374}]) [2025-01-08 00:51:11.033305] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B9B-0-0] [lt=11] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:11.068907] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C654-0-0] [lt=1][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963663690 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55716} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4746} write_epoch=1, prev=this=0x2ad8c146d618 trans_version={val:1736322701445233000, v:0} scn={val:1736322701445233000, v:0} tx_id={txid:50203} prev=0x2ad8c146d460 next=(nil) modify_count=2 acc_checksum=133928624 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:2410} write_epoch=3, snapshot_version={val:1736326271065652000, v:0}, *this={this=0x2ad8c14594d0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad8c146d618 latest_compact_node=(nil) max_trans_version={val:1736326208860596000, v:0} max_trans_id=55610 max_elr_trans_version={val:1736326208860596000, v:0} max_elr_trans_id=55610 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=3 max_modify_scn={val:1736322701445233000, v:0} min_modify_scn={val:1736322641305035000, v:0}}) [2025-01-08 00:51:11.069051] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C654-0-0] [lt=142][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963663690 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55716} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4746} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300964889, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300964889, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326271065652000, v:0}, tx_id:{txid:55716}, scn:{branch:0, seq:3578}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55716}, tx_desc:{this:0x2ad8b08923e0, tx_id:{txid:55716}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326271068504, commit_ts:-1, finish_ts:-1, timeout_us:29999988, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326271155525, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:4746}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:11.069232] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C654-0-0] [lt=178][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55716}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4746}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:11.069294] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C654-0-0] [lt=60][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55716} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:11.069327] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C654-0-0] [lt=31][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:11.069341] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C654-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 463D9BF62D2B0600, int: 1736326271155526}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 463D9BF62D2B0600, int: 1736326271155526}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 00D0CF0000000000, int: 13619200}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00D0CF0000000000, int: 13619200}]}) [2025-01-08 00:51:11.069874] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C654-0-0] [lt=112][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:11.072795] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C654-0-0] [lt=131] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8257}, to_seq_no={branch:0, seq:3579}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55716} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:11.098891] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C656-0-0] [lt=17][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963663860 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55717} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7110} write_epoch=1, prev=this=0x2ad8c1567138 trans_version={val:1736324387340138000, v:0} scn={val:1736324387340138000, v:0} tx_id={txid:52612} prev=(nil) next=(nil) modify_count=0 acc_checksum=3727195837 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:6208} write_epoch=1, snapshot_version={val:1736326271089600000, v:0}, *this={this=0x2ad8c15670a8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1567138 latest_compact_node=(nil) max_trans_version={val:1736326208906272000, v:0} max_trans_id=55611 max_elr_trans_version={val:1736326208906272000, v:0} max_elr_trans_id=55611 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324387340138000, v:0} min_modify_scn={val:1736324387340138000, v:0}}) [2025-01-08 00:51:11.099102] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C656-0-0] [lt=162][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963663860 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55717} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7110} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326300991409, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326300991409, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326271089600000, v:0}, tx_id:{txid:55717}, scn:{branch:0, seq:7107}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55717}, tx_desc:{this:0x2ad8b0894440, tx_id:{txid:55717}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326271098307, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326271180005, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:7110}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:11.099768] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C656-0-0] [lt=654][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55717}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:7110}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:11.099820] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C656-0-0] [lt=49][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55717} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:11.099909] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C656-0-0] [lt=85][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:11.099940] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C656-0-0] [lt=24][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 76A19BF62D2B0600, int: 1736326271181174}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 76A19BF62D2B0600, int: 1736326271181174}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 00E0010000000000, int: 122880}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00E0010000000000, int: 122880}]}) [2025-01-08 00:51:11.100353] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C656-0-0] [lt=317][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:11.104265] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:11.108059] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C656-0-0] [lt=45] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:16722}, to_seq_no={branch:0, seq:7108}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55717} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:11.110553] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=160] ====== check clog disk timer task ====== [2025-01-08 00:51:11.110596] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=33] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:11.114967] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=101] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=480, get_gts_with_stc_cnt=72, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=70, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:11.115099] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=82] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=421, get_gts_with_stc_cnt=54, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=54, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:11.117502] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.124333] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:11.125908] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2184773}], scn_array=[cnt:1, {val:1736326271088513000, v:0}], curr_size=122, accum_size=366, time_cost=20460) [2025-01-08 00:51:11.134358] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:11.138137] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58438][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9167-0-0] [lt=98] init point thread id with(&point=0x5572d50e3ac0, idx_=1094, point=[thread id=58438, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58438) [2025-01-08 00:51:11.145927] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B5BBA66-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1736326271144478000, start_id=1736326271144478000, end_id=1736326271144478001) [2025-01-08 00:51:11.160711] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84282646}], scn_array=[cnt:1, {val:1736326271140669000, v:0}], curr_size=562, accum_size=11163, time_cost=13908) [2025-01-08 00:51:11.162125] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:11.163288] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18668656}], scn_array=[cnt:1, {val:1736326271144478000, v:0}], curr_size=545, accum_size=1033, time_cost=14862) [2025-01-08 00:51:11.180762] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C658-0-0] [lt=37][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963663a30 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:18839} write_epoch=1, prev=this=0x2ad9636561d0 trans_version={val:1736326208954439000, v:0} scn={val:1736326208954439000, v:0} tx_id={txid:55612} prev=0x2ad8dae89320 next=(nil) modify_count=60 acc_checksum=1008754160 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:29067} write_epoch=3, snapshot_version={val:1736326271172464000, v:0}, *this={this=0x2ad8c1567480 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad9636561d0 latest_compact_node=0x2ad8dae89320 max_trans_version={val:1736326208954439000, v:0} max_trans_id=55612 max_elr_trans_version={val:1736326208954439000, v:0} max_elr_trans_id=55612 latest_compact_ts=1736326208971605 last_compact_cnt=59 total_trans_node_cnt=61 max_modify_scn={val:1736326208954439000, v:0} min_modify_scn={val:1736324387368277000, v:0}}) [2025-01-08 00:51:11.180913] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C658-0-0] [lt=149][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963663a30 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:18839} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326301072950, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326301072950, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326271172464000, v:0}, tx_id:{txid:55718}, scn:{branch:0, seq:18836}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55718}, tx_desc:{this:0x2ad8b089e620, tx_id:{txid:55718}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326271179558, commit_ts:-1, finish_ts:-1, timeout_us:29999972, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326271247725, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:18839}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:11.181067] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C658-0-0] [lt=152][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55718}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:18839}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:11.181094] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C658-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55718} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:11.181131] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C658-0-0] [lt=33][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:11.181143] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C658-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 45D69CF62D2B0600, int: 1736326271260229}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 45D69CF62D2B0600, int: 1736326271260229}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 00D0240000000000, int: 2412544}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00D0240000000000, int: 2412544}]}) [2025-01-08 00:51:11.181345] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C658-0-0] [lt=118][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:11.184026] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C658-0-0] [lt=103] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:27494}, to_seq_no={branch:0, seq:18837}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55718} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:11.204345] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=82] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2188991, this={dio_aligned_buf:{buf_write_offset:1849, buf_padding_size:2247, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:23916, truncate_used_ts:5443}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2187264, buf_len=122, total_write_size=7658, total_write_size_after_dio=258048, ob_pwrite_used_ts=1532357, count=61, each_pwrite_cost=25120) [2025-01-08 00:51:11.217412] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65C-0-0] [lt=13][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963664050 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55720} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4612} write_epoch=1, prev=this=0x2ad8c15677a8 trans_version={val:1736324387413431000, v:0} scn={val:1736324387413431000, v:0} tx_id={txid:52615} prev=(nil) next=(nil) modify_count=0 acc_checksum=643497365 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:2541} write_epoch=1, snapshot_version={val:1736326271214365000, v:0}, *this={this=0x2ad8c1567718 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c15677a8 latest_compact_node=(nil) max_trans_version={val:1736326209107439000, v:0} max_trans_id=55614 max_elr_trans_version={val:1736326209107439000, v:0} max_elr_trans_id=55614 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324387413431000, v:0} min_modify_scn={val:1736324387413431000, v:0}}) [2025-01-08 00:51:11.217573] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.217503] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65C-0-0] [lt=89][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963664050 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55720} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4612} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326301113697, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326301113697, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326271214365000, v:0}, tx_id:{txid:55720}, scn:{branch:0, seq:4609}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55720}, tx_desc:{this:0x2ad8b08a67a0, tx_id:{txid:55720}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326271216683, commit_ts:-1, finish_ts:-1, timeout_us:29999963, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326271303720, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:4612}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:11.217608] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65C-0-0] [lt=104][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55720}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4612}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:11.217626] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65C-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55720} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:11.217660] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65C-0-0] [lt=31][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:11.217673] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65C-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 73849DF62D2B0600, int: 1736326271304819}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 73849DF62D2B0600, int: 1736326271304819}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 0000BA0000000000, int: 12189696}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 0000BA0000000000, int: 12189696}]}) [2025-01-08 00:51:11.217864] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65C-0-0] [lt=59][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:11.219989] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C65C-0-0] [lt=107] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7049}, to_seq_no={branch:0, seq:4610}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55720} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:11.233970] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:11.234124] WDIAG read_multi_row (ob_table_access_helper.h:238) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:11.236376] INFO delete_row (ob_table_access_helper.h:339) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326271234142) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326271234142) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:11.236462] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=28] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:11.236520] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:51:11.236571] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:51:11.241288] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65E-0-0] [lt=33][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad9636642d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55721} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7116} write_epoch=1, prev=this=0x2ad8c156f2f0 trans_version={val:1736324448441416000, v:0} scn={val:1736324448441416000, v:0} tx_id={txid:52738} prev=0x2ad8c156f138 next=(nil) modify_count=2 acc_checksum=2139832314 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:16922} write_epoch=3, snapshot_version={val:1736326271237859000, v:0}, *this={this=0x2ad8c1567908 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad8c156f2f0 latest_compact_node=(nil) max_trans_version={val:1736326209159811000, v:0} max_trans_id=55615 max_elr_trans_version={val:1736326209159811000, v:0} max_elr_trans_id=55615 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=3 max_modify_scn={val:1736324448441416000, v:0} min_modify_scn={val:1736324387460173000, v:0}}) [2025-01-08 00:51:11.241354] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65E-0-0] [lt=64][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad9636642d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55721} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7116} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326301134342, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326301134342, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326271237859000, v:0}, tx_id:{txid:55721}, scn:{branch:0, seq:7113}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55721}, tx_desc:{this:0x2ad8b08aa860, tx_id:{txid:55721}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326271240132, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326271324038, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:7116}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:11.241450] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65E-0-0] [lt=93][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55721}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:7116}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:11.241467] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65E-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55721} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:11.241499] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65E-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:11.241512] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65E-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 86D89DF62D2B0600, int: 1736326271326342}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 86D89DF62D2B0600, int: 1736326271326342}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 0060870000000000, int: 8871936}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 0060870000000000, int: 8871936}]}) [2025-01-08 00:51:11.241620] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D17C9C65E-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:11.242891] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C65E-0-0] [lt=100] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10755}, to_seq_no={branch:0, seq:7114}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55721} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:11.251248] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C65F-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1736326271250369000, start_id=1736326271250369000, end_id=1736326271250369001) [2025-01-08 00:51:11.254563] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=76] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:11.261438] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1002}, {id:1001}]) [2025-01-08 00:51:11.261929] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C660-0-0] [lt=55][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8b658 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:85} seq_no={branch:0, seq:5787} write_epoch=1, prev=this=0x2ad8c1459938 trans_version={val:1736322641365392000, v:0} scn={val:1736322641365392000, v:0} tx_id={txid:50120} prev=(nil) next=(nil) modify_count=0 acc_checksum=2274121886 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326269868685000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:85} seq_no={branch:0, seq:28846} write_epoch=1, snapshot_version={val:1736326271259508000, v:0}, *this={this=0x2ad8c14598a8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1459938 latest_compact_node=(nil) max_trans_version={val:1736326209195385000, v:0} max_trans_id=55616 max_elr_trans_version={val:1736326209195385000, v:0} max_elr_trans_id=55616 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641365392000, v:0} min_modify_scn={val:1736322641365392000, v:0}}) [2025-01-08 00:51:11.262177] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C660-0-0] [lt=245][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8b658 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:85} seq_no={branch:0, seq:5787} write_epoch=1, ctx={this:0x2ad8ad351678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326301157372, tablet_id:{id:309}, table_iter:0x2ad8ad34fb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326301157372, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326271259508000, v:0}, tx_id:{txid:55722}, scn:{branch:0, seq:5784}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55722}, tx_desc:{this:0x2ad8b5c53840, tx_id:{txid:55722}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326271260796, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326271346513, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:5787}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:11.262320] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C660-0-0] [lt=141][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55722}, data:0x2ad8ad34e508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:5787}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:11.262341] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C660-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55722} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:11.262434] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C660-0-0] [lt=91][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:11.262449] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C660-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96580c6a0, local_datums:0x2ad96580c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96580c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96580c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96580c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96580c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96580c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96580c372, hex: 302C9EF62D2B0600, int: 1736326271347760}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96580c37a, hex: 302C9EF62D2B0600, int: 1736326271347760}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96580c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96580c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:11.262542] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C660-0-0] [lt=60][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:11.264230] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C660-0-0] [lt=55] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9295}, to_seq_no={branch:0, seq:5785}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55722} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:11.265764] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:51:11.265936] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:11.269951] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C9BB5CB-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326271267895000, start_id=1736326271267895000, end_id=1736326271267895001) [2025-01-08 00:51:11.272004] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46388][T1_L0_G0][T1][YB42C0A89F8B-00062B2D170BBD9C-0-0] [lt=69] get number(ret=0, service_type_=0, range=1, base_id=1736326271267895000, start_id=1736326271267895000, end_id=1736326271267895001) [2025-01-08 00:51:11.272539] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=193] ====== check clog disk timer task ====== [2025-01-08 00:51:11.272575] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:11.279093] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46508][T1_L0_G0][T1][YB42C0A89F8A-00062B2D17C9C661-0-0] [lt=53] get number(ret=0, service_type_=0, range=1, base_id=1736326271277944000, start_id=1736326271277944000, end_id=1736326271277944001) [2025-01-08 00:51:11.282808] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:11.283051] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:11.287156] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3A4-0-0] [lt=35][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:11.291237] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=70] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:11.306417] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=71] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:11.309468] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5990) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=48] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000) [2025-01-08 00:51:11.309565] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=96] get multi version start(ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.309648] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.309826] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=66] get multi version start(ls_id={id:1}, tablet_id={id:50380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.309895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.309957] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310085] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:50192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310274] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60429}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310537] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310603] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310664] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310725] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310782] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310843] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.310903] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.311163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=75] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846994}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.311317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=96] get multi version start(ls_id={id:1}, tablet_id={id:101109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.311379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50124}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.311370] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:51:11.311439] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.311500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.311560] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.311736] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.311934] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312249] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312378] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101098}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312439] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312496] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312675] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312735] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101036}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.312964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101064}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60129}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313207] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313414] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=126] get multi version start(ls_id={id:1}, tablet_id={id:101080}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313796] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.313979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101083}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314101] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314244] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101062}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314431] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:129}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314492] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101018}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.314959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101034}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315265] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315332] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101084}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315445] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315511] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315572] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50219}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315634] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315694] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315755] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315816] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315873] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101068}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315934] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.315994] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316055] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60140}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316176] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50484}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316433] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316602] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316698] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316760] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=2079744) [2025-01-08 00:51:11.316769] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.316993] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317055] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317240] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317424] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101073}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317491] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317553] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:128}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317746] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.317870] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101099}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318282] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101092}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318843] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60484}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318918] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.318986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.319045] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101031}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.319108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.319169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.319306] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:11.319759] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.319761] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=70] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.319926] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.321006] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846983}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321226] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=105] get multi version start(ls_id={id:1}, tablet_id={id:60425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321363] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321424] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846990}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321418] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A916C-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:11.321485] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321547] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321670] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321799] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321861] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.321983] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322044] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322294] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=83] get multi version start(ls_id={id:1}, tablet_id={id:50292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322357] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50140}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322537] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101067}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:429}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.322959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=58] get multi version start(ls_id={id:1}, tablet_id={id:60309}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323135] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101027}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323248] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846986}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323467] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=64] get multi version start(ls_id={id:1}, tablet_id={id:60103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846977}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.323950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324043] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324440] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324624] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101082}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324683] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324744] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324805] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324927] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.324988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325048] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325109] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325148] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:11.325170] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325472] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325533] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325594] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60502}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200009}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.325956] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326078] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60116}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326199] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326342] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101057}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326404] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326465] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326526] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326769] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326829] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.326951] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327012] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327073] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101033}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327193] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327339] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=89] get multi version start(ls_id={id:1}, tablet_id={id:60251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327402] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327585] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327646] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101066}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327707] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327768] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327829] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:320}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.327951] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846980}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328012] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328195] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328436] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328497] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101030}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101094}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101020}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328798] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50117}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328860] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101044}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.328934] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329000] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329123] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329184] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329341] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329412] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329536] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50395}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:118}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329897] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.329958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.330019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.330076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:445}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.330136] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101017}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.330263] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.330307] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO [ratelimit] time: 1736326271330304, bytes: 872721342, bw: 1.051629 MB/s, add_ts: 1001604, add_bytes: 1104482 [2025-01-08 00:51:11.330331] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101013}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.330392] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.331149] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18669201}], scn_array=[cnt:1, {val:1736326271144478001, v:0}], curr_size=122, accum_size=1155, time_cost=11591) [2025-01-08 00:51:11.331319] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=4695, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:11.331338] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2197426}], scn_array=[cnt:1, {val:1736326271084144002, v:0}], curr_size=122, accum_size=976, time_cost=11443) [2025-01-08 00:51:11.331352] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=976) [2025-01-08 00:51:11.331363] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=374, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:11.331382] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=4729, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:11.331396] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2164024}], scn_array=[cnt:1, {val:1736326271144478000, v:0}], curr_size=122, accum_size=1098, time_cost=11738) [2025-01-08 00:51:11.331409] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:11.331469] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=57] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=630, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:11.331623] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=104] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=95, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:11.331662] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:11.331672] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=8831, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:11.331682] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=19, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:11.331755] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:11.332015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332092] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332292] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332362] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60256}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101010}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332660] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101072}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332721] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332782] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332843] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.332904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101088}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333091] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101069}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333271] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333335] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333457] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333518] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333578] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333639] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333700] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333761] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.333943] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50128}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101060}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334267] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50256}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334335] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334397] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101100}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334458] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334519] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334642] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334764] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334826] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101022}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.334947] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335037] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=115] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:11.335130] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335241] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335431] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335493] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335554] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335916] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.335977] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336038] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336099] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336290] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101035}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336414] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:5}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336537] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336659] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101058}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101081}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.336960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101086}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101052}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337301] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101019}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101079}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337783] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337978] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.337972] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAD-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:11.338001] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAD-0-0] [lt=26] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:11.338038] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338246] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338314] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338493] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338553] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338614] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60320}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338853] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.338913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200010}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339295] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=80] get multi version start(ls_id={id:1}, tablet_id={id:50480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339419] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339479] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339603] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339656] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1B49C747-0-0] [lt=24] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=976) [2025-01-08 00:51:11.339698] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1B49C747-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=21, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:11.339712] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1B49C747-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=20389, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:11.339720] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1B49C747-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=61, avg_log_submit_to_flush_time=5125, avg_log_submit_to_slide_time=20327) [2025-01-08 00:51:11.339775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339843] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.339964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340079] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340262] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:200006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340390] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340450] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340511] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340757] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340877] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.340936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.341661] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=976, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:11.341697] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17962, miss_cnt:0, cache_read_size:2197548, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:11.344109] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.344663] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=360] get multi version start(ls_id={id:1}, tablet_id={id:397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.344813] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=85] get multi version start(ls_id={id:1}, tablet_id={id:110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.344881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345037] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345120] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:60146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345184] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345370] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345494] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345554] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345676] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345736] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345918] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:155}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.345978] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50309}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346038] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346277] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346353] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346473] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:453}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346759] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346816] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50125}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346873] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846979}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.346988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347045] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347103] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347160] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347291] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347355] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347413] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347680] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347852] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.347966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348024] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348135] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101043}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348241] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348305] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348421] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348479] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348536] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348594] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348765] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348823] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348938] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.348995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846988}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=81] get multi version start(ls_id={id:1}, tablet_id={id:446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349359] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101032}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349474] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101089}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:200008}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349701] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:124}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.349964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350022] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:200005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350263] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350386] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350501] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60128}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350559] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350843] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350900] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.350957] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351104] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351260] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=56] get multi version start(ls_id={id:1}, tablet_id={id:60236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351322] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351438] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351495] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351550] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:140}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351603] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:200001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351656] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101028}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351827] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351884] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.351999] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352245] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50155}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352369] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101021}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352599] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352658] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352830] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352888] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.352945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353002] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353242] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353369] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353610] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50116}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353784] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101037}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.353955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354013] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354070] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354182] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101054}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354239] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354425] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354483] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60395}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354656] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354824] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50123}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101041}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.354995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.355076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:200012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.355134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.358086] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:11.358114] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:11.363385] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=20] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:11.363966] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=46] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:11.364541] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=22] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:11.364627] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=18] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326264299814000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:11.365959] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=18] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:11.365975] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185422-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:11.366048] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=18] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326265706032000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:11.366532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.366625] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:101097}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.366690] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.366751] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.366858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.366930] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101024}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.366990] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367283] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367391] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367564] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:483}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.367941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101055}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368002] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368124] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368181] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368329] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368445] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368506] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:123}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368680] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=57] get multi version start(ls_id={id:1}, tablet_id={id:50290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368742] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368802] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368923] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:454}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.368983] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369102] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101063}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369297] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369362] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:3}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369600] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:219}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369721] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846985}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369897] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:141}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.369957] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370016] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101040}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370294] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846978}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:50494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370531] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101016}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370653] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370701] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=89] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:11.370776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370853] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101047}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.370950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371014] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371072] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371191] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371368] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371445] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371513] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846993}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371578] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371640] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371701] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371763] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371824] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101059}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371884] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.371945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372066] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372187] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372318] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60219}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372444] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372506] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372662] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60155}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.372967] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373122] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=99] get multi version start(ls_id={id:1}, tablet_id={id:206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373185] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=202] get multi version start(ls_id={id:1}, tablet_id={id:498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373605] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373791] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373852] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.373974] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374035] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374095] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374280] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374467] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374646] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101046}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374836] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374897] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.374957] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.375018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.375079] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.375140] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.375197] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.375376] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.375445] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:117}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.375507] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.376447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.376530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:50394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.376591] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.376735] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.376903] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=109] get multi version start(ls_id={id:1}, tablet_id={id:251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.376997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.377087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.377175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.377341] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=108] get multi version start(ls_id={id:1}, tablet_id={id:50291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.377435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101045}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.377583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=92] get multi version start(ls_id={id:1}, tablet_id={id:50371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.377674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.377763] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:502}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.377850] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.377938] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.378026] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.378114] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60453}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.378293] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=122] get multi version start(ls_id={id:1}, tablet_id={id:131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.378544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.378637] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.378726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.378887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.378979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.379064] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101015}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.379306] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=95] get multi version start(ls_id={id:1}, tablet_id={id:246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.379587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=216] get multi version start(ls_id={id:1}, tablet_id={id:60499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.379666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:101078}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.379730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.379792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.379853] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.379914] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.379974] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101049}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380145] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380455] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380517] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8707F-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:11.380531] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380567] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:11.380579] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:11.380596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101101}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380659] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380721] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380782] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846995}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380938] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.380999] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381248] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381440] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:256}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381614] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381748] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101091}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381806] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101008}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.381987] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101039}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382106] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382164] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101029}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382425] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382485] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382546] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382606] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382848] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.382969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383299] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383573] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383641] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383763] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.383978] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384037] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384284] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384351] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384413] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384473] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50129}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384534] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101095}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101070}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384835] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.384955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101075}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385253] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385339] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385403] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846987}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385501] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58435][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B98-0-0] [lt=43] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=27135) [2025-01-08 00:51:11.385524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101077}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385585] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385705] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60118}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385765] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385825] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101071}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.385942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101093}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386003] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386239] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386308] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386369] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101065}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386495] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386553] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:309}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386733] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386848] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.386967] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387092] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387156] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:116}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387290] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387351] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:100003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387412] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387590] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101014}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.387968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.388027] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.388088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.388148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.388273] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.388345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.388406] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.388466] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.388560] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:101004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389496] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389733] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389853] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.389968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390281] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=127] get multi version start(ls_id={id:1}, tablet_id={id:50221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390404] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390521] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390582] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50445}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.390942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391002] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846981}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391122] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101050}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391250] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391319] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391438] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391558] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391619] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60124}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391799] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50429}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391920] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.391980] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50453}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392040] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392099] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101056}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392285] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392590] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.392889] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50154}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393016] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:101023}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393269] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:101007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393332] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101048}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393453] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60445}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101026}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393570] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393691] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50141}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393750] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393810] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393870] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393930] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.393990] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394049] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394109] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=103] get multi version start(ls_id={id:1}, tablet_id={id:60335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394412] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394599] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394660] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101025}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.394961] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395022] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60141}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395079] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395139] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395196] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=98] get multi version start(ls_id={id:1}, tablet_id={id:487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395434] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:4}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:100005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395799] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846996}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395919] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.395979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396039] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396099] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396158] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396306] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396434] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846991}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396492] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396552] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396612] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396732] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396791] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396910] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60117}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.396971] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101085}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397286] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397354] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50454}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101087}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.397954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60123}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398014] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398073] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398257] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50483}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398325] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398446] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398506] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398566] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:484}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398747] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.398923] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.399015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:101051}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.399077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846982}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.399134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.399194] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.399932] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846984}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400072] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400132] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400251] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400321] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400382] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400503] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101061}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400623] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400683] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400779] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101076}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.400964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401024] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60454}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401284] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401395] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401459] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401580] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401640] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401700] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401761] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401821] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.401998] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:154}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402114] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402220] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402287] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846989}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101074}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402531] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402591] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402652] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101053}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.402952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=13] REACH SYSLOG RATE LIMIT [frequency] [2025-01-08 00:51:11.404896] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=27] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:11.406408] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:11.406597] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326265595928000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:11.407568] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=0] get multi version start(ls_id={id:1}, tablet_id={id:101110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324465706032000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324455686329000, max_merged_snapshot=1) [2025-01-08 00:51:11.407806] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:116) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E1-0-0] [lt=0] SSTableGCTask(cost_ts=98801) [2025-01-08 00:51:11.416606] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=109] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=37, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:11.416651] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:11.416696] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1098) [2025-01-08 00:51:11.416713] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:11.420080] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.420128] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8535-0-0] [lt=66] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1643) [2025-01-08 00:51:11.422044] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:11.424516] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=50, stat_interval=1000000, avg (count/cost)=371, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:11.424542] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=50, stat_interval=1000000, avg (count/cost)=7, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:11.424708] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2197604}, scn={val:1736326271084144003, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:11.424816] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=214] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324465595928000) [2025-01-08 00:51:11.424985] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=165] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2197604}, scn_:{val:1736326271084144003, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326271431493, first_handle_ts_:1736326271424802, replay_cost_:179, retry_cost_:-1, read_log_buf_:0x2ad94da04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326271424041, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2197660}, next_to_submit_scn_:{val:1736326271084144004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2197548}, end_lsn:{lsn:2197660}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2197660}, readable_log_tail:{lsn:2197660}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64907108, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326271084144003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326271084144003, v:0}, data_checksum:302994773, flag:11867}}, init_mode_version:0, accumulate_checksum:3240838010, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326271084144003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17963, miss_cnt:0, cache_read_size:2197660, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=5, ret=0) [2025-01-08 00:51:11.427887] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D03E-0-0] [lt=53][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:11.432024] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=28] ====== tenant freeze timer task ====== [2025-01-08 00:51:11.433227] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=28] gc stale ls task succ [2025-01-08 00:51:11.435689] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16] ====== tenant freeze timer task ====== [2025-01-08 00:51:11.435765] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=70] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71124160, tenant_id=1001) [2025-01-08 00:51:11.435787] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:11.459163] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.459293] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.470705] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=51] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:11.470789] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=76] temporary file module use no blocks [2025-01-08 00:51:11.470803] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:11.470817] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:11.470830] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193775384401, flush_io_finished_ret_:0, flush_io_finished_round_:1930, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:11.470885] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=45] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:11.470899] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] temporary file module use no blocks [2025-01-08 00:51:11.470905] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:11.470914] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:11.470924] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193775384506, flush_io_finished_ret_:0, flush_io_finished_round_:3758, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:11.516765] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=131] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:11.516805] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:11.519325] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=269] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.520414] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.525291] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:11.535533] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:11.539731] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=61] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=26569) [2025-01-08 00:51:11.539772] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND COST TIME](cur_stat_count=41, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:11.541632] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=48] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.542109] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:11.542161] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4726, iops:39, bw:182847, [delay/us]:prepare:14, schedule:172, submit:222, rt:2422, total:2831) [2025-01-08 00:51:11.542220] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:11.542290] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4726, avg_iops:39, avg_bw:182863, [delay/us]: prepare:14, schedule:172, submit:222, device:2422, total:2831) [2025-01-08 00:51:11.542304] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:11.542353] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4864, iops:16, bw:77211, [delay/us]:prepare:27, schedule:156, submit:700, rt:2448, total:3332) [2025-01-08 00:51:11.542365] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:11.542403] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4864, avg_iops:16, avg_bw:77213, [delay/us]: prepare:27, schedule:156, submit:700, device:2448, total:3332) [2025-01-08 00:51:11.542413] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:11.542435] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20319, [delay/us]:prepare:7, schedule:561, submit:1, rt:5999, total:6570) [2025-01-08 00:51:11.542457] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:162554, [delay/us]:prepare:11, schedule:321, submit:258, rt:3958, total:4550) [2025-01-08 00:51:11.542466] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:11.542560] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=88] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20318, [delay/us]: prepare:7, schedule:561, submit:1, device:5999, total:6570) [2025-01-08 00:51:11.542577] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:162544, [delay/us]: prepare:11, schedule:321, submit:258, device:3958, total:4550) [2025-01-08 00:51:11.542588] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:11.542609] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4063, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:297, total:305) [2025-01-08 00:51:11.542635] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:11.542662] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4063, [delay/us]: prepare:0, schedule:0, submit:600000000, device:297, total:305) [2025-01-08 00:51:11.542681] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:11.542696] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:11.542704] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:11.542712] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:11.542720] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=958kB/s, net_out=385kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:11.544026] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=61] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.549953] INFO [SERVER] refresh_avail_memory (ob_table_load_assigned_memory_manager.cpp:117) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=20] ObTableLoadAssignedMemoryManager::refresh_avail_memory(MTL_ID()=1, avail_memory=107374182, chunk_count_=0, avail_sort_memory_=107374182, avail_memory_=107374182) [2025-01-08 00:51:11.550888] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=7] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:15002371794, tenant_id:1}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:11.551033] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=49] recycle one thread(this=0x2ad8881fc3b0, total=7, remain=6) [2025-01-08 00:51:11.551050] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:15002371794, tenant_id:1}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:11.551063] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7764590743, tenant_id:1001}, cur_thread_count=8, cnt=-1, new_thread_count=7) [2025-01-08 00:51:11.551476] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] recycle one thread(this=0x2ad8f90b0e70, total=8, remain=7) [2025-01-08 00:51:11.551500] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=23] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7764590743, tenant_id:1001}, cur_thread_count=8, cnt=-1, new_thread_count=7) [2025-01-08 00:51:11.577159] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C96-0-0] [lt=100][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:11.578479] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C96-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:11.588856] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C62A-0-0] [lt=62][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193776502689, cache_obj->added_lc()=false, cache_obj->get_object_id()=13609, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:11.604380] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3815-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:11.604427] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3815-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:11.604446] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3815-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:11.607841] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BEC8-0-0] [lt=11] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:11.616037] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1736326271615157000, start_id=1736326271615157000, end_id=1736326271615157001) [2025-01-08 00:51:11.616160] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=91] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:11.617665] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58435][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD89-0-0] [lt=23] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326271709972, keepalive_ts:-1}) [2025-01-08 00:51:11.617783] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58435][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD89-0-0] [lt=116] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326271709972, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:11.618464] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4573-0-0] [lt=67] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:51:11.623723] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.632468] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B9D-0-0] [lt=17] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:11.675465] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=45] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:11.675491] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:11.675624] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84981-0-0] [lt=25] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:11.675720] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84981-0-0] [lt=92] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:11.703620] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3817-0-0] [lt=54] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:11.703668] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3817-0-0] [lt=47] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38577, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:11.703680] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3817-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=5194, avg_log_submit_to_slide_time=38555) [2025-01-08 00:51:11.703859] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:11.703948] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17865, miss_cnt:0, cache_read_size:2185505, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:11.717164] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=103, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:11.717325] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:51:11.717616] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=37] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:392K/90:227628 disk:392K/90:227628 t1m2:132K/29:179133 t1m2g0:132K/29:179133 t1002m2:188K/45:286085 t1002m2g0:188K/45:286085 t1001m2:72K/16:151116 t1001m2g0:72K/16:151116 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:11.718563] INFO [LIB] stat (utility.h:1210) [58438][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4575-0-0] [lt=143] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:11.719098] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=147] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5797, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:11.719139] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:11.719161] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=655, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:11.719323] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=36, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:11.721042] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8540-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:11.721078] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8540-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9069, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:11.723845] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=130][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.726437] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=12] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1211056, replace_num=15728) [2025-01-08 00:51:11.726469] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:11.736158] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:11.736802] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50896][T1002_L0_G2][T1002][YB42C0A89F8A-00062B2D1E79E49F-0-0] [lt=24] get_min_unreplayed_log_info(lsn={lsn:2198148}, scn={val:1736326271511167003, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326271721724, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2198148}, next_to_submit_scn_:{val:1736326271511167003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2198026}, end_lsn:{lsn:2198148}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2198148}, readable_log_tail:{lsn:2198148}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64906620, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326271511167002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326271511167002, v:0}, data_checksum:3348837161, flag:7966}}, init_mode_version:0, accumulate_checksum:1041351354, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326271511167002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17967, miss_cnt:0, cache_read_size:2198148, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:11.740365] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=29, stat_interval=1000000, avg (count/cost)=85, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:11.740408] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=29, avg_log_batch_cnt=1, total_group_log_size=16847, avg_group_log_size=580) [2025-01-08 00:51:11.741412] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=133] [PALF STAT WRITE LOG COST TIME](cur_stat_count=29, stat_interval=1000000, avg (count/cost)=3149, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:11.741438] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=16847) [2025-01-08 00:51:11.741456] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=29, stat_interval=1000000, avg (count/cost)=686, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:11.741482] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FLUSH CB COST TIME](cur_stat_count=29, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:11.743151] INFO [LIB] stat (utility.h:1210) [58435][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B9B-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=29, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:11.743173] INFO [LIB] stat (utility.h:1210) [58435][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B9B-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=29, stat_interval=1000000, avg (count/cost)=6554, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:11.743258] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58435][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5B9B-0-0] [lt=83] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=29, avg_log_gen_to_freeze_time=319, avg_log_gen_to_submit_time=425, avg_log_submit_to_flush_time=4020, avg_log_submit_to_slide_time=6129) [2025-01-08 00:51:11.743481] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=16847, hit_cnt=30, read_cnt=30, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:11.751686] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5225452946, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:11.751823] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31] recycle one thread(this=0x2ad8885abcf0, total=5, remain=4) [2025-01-08 00:51:11.751840] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5225452946, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:11.762070] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE842-0-0] [lt=54] get number(ret=0, service_type_=0, range=1, base_id=1736326271761908000, start_id=1736326271761908000, end_id=1736326271761908001) [2025-01-08 00:51:11.767593] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50497][T1002_L5_G0][T1002][YB42C0A89F8A-00062B2D1B09C70A-0-0] [lt=74] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=2.703295364379882812e+02, this=0x2ad8f30a3208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1535, Resource Limit(MB)=1351, Throttle Trigger(MB)=810, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=5.353173586556075669e+00) [2025-01-08 00:51:11.772026] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=71] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=136, avg_log_submit_to_flush_time=6313, avg_log_submit_to_slide_time=8482) [2025-01-08 00:51:11.772149] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2188, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:11.772751] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:11.773993] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1736326271773990, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006974, add_bytes: 0 [2025-01-08 00:51:11.774127] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58435][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=13] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:51:11.783301] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:11.783459] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:11.784825] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=107] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326271375286001, v:0}, new_max_version={val:1736326271375286001, v:0}, last_min_version={val:1736326271277944000, v:0}, last_max_version={val:1736326271277944000, v:0}, total_time_us=5327, delta_us=409536) [2025-01-08 00:51:11.784859] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326271375286001, v:0}, min_version={val:1736326271375286001, v:0}, max_version={val:1736326271375286001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:11.787592] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=44] ====== check clog disk timer task ====== [2025-01-08 00:51:11.787759] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=119] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:11.792286] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=150] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:11.806556] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:11.813977] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:11.815076] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58385][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=27] worker thread exit(tenant_->id()=1001, workers_.get_size()=3) [2025-01-08 00:51:11.816911] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:11.819615] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=52] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:11.821570] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1769CA50-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326271818765000, start_id=1736326271818765000, end_id=1736326271818765001) [2025-01-08 00:51:11.824275] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=71, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:11.824352] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.825780] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=3662, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:11.825806] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10491) [2025-01-08 00:51:11.825841] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=1187, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:11.826435] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58437][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6835-0-0] [lt=34] init point thread id with(&point=0x5572d50e3a80, idx_=1093, point=[thread id=58437, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58437) [2025-01-08 00:51:11.827487] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10652) [2025-01-08 00:51:11.827529] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=26, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:11.828964] INFO [LIB] stat (utility.h:1210) [58434][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA391F-0-0] [lt=55] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=23, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:11.828999] INFO [LIB] stat (utility.h:1210) [58434][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA391F-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=7071, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:11.829012] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58434][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA391F-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=176, avg_log_gen_to_submit_time=260, avg_log_submit_to_flush_time=5320, avg_log_submit_to_slide_time=6811) [2025-01-08 00:51:11.829330] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10491, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:11.829484] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=36] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:11.829523] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:11.829547] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=15] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=178) [2025-01-08 00:51:11.829515] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=22] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:11.829577] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=60] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1) [2025-01-08 00:51:11.837020] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F6-0-0] [lt=32] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:11.837082] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F6-0-0] [lt=48] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326271760551000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326271760551000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:11.849638] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F6-0-0] [lt=116] get number(ret=0, service_type_=0, range=1, base_id=1736326271848619000, start_id=1736326271848619000, end_id=1736326271848619001) [2025-01-08 00:51:11.849932] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F6-0-0] [lt=50] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=11241, avg_group_log_size=591) [2025-01-08 00:51:11.849973] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F6-0-0] [lt=36] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11241) [2025-01-08 00:51:11.853377] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=18] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:11.867071] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:11.868788] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=155] get_max_applied_scn(scn={val:1736326271767610000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18670356}, palf_committed_end_scn_:{val:1736326271767610000, v:0}, last_check_scn_:{val:1736326271767610000, v:0}, max_applied_cb_scn_:{val:1736326271767610000, v:0}}) [2025-01-08 00:51:11.868826] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18670356}, readable_log_tail:{lsn:18670356}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48434412, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:11.868927] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=76] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:11.869039] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=10] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:11.896868] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.896905] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.896931] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.896954] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:11.905552] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=96] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3766, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:11.905581] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:11.905601] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=156, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:11.905714] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=64] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=56, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:11.919939] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4579-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18060, log_task={header:{begin_lsn:{lsn:2198148}, end_lsn:{lsn:2198270}, log_id:18060, min_scn:{val:1736326271511167003, v:0}, max_scn:{val:1736326271511167003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2198026}, prev_proposal_id:1, committed_end_lsn:{lsn:2198148}, data_checksum:1715074492, accum_checksum:2540093391, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326271819797, freeze_ts:1736326271819797, submit_ts:1736326271819803, flushed_ts:1736326271824618, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:4815}, fs_cb_begin_ts=1736326271919930, log_life_time=100133) [2025-01-08 00:51:11.924513] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:11.926659] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=124] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:11.926893] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:11.926919] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58438][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=72] worker thread exit(tenant_->id()=1002, workers_.get_size()=5) [2025-01-08 00:51:11.930126] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=44] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8633, offset=2449887}, end_cursor:ObLogCursor{file_id=1, log_id=8634, offset=2450192}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947d4050, len:3617, log_data_len:305, seq:8633, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:11.930610] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4D-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:11.930650] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4D-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326271767610000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326271767610000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:11.937449] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=56] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:11.939860] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=50] LogLoopThread round_cost_time(us)(round_cost_time=13) [2025-01-08 00:51:11.940864] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=53] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8634, offset=2450192}, end_cursor:ObLogCursor{file_id=1, log_id=8635, offset=2450489}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947d8050, len:3312, log_data_len:297, seq:8634, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:11.941410] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4D-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:11.941449] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4D-0-0] [lt=22] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326271767610000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326271767610000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:11.952053] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15074302072, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:11.952341] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=32] recycle one thread(this=0x5572d527f900, total=5, remain=4) [2025-01-08 00:51:11.952378] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=36] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15074302072, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:11.952437] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=59] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7799040901, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:11.952501] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] recycle one thread(this=0x2ad8c0eb0e70, total=6, remain=5) [2025-01-08 00:51:11.952542] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=39] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7799040901, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:11.952699] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=55] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8635, offset=2450489}, end_cursor:ObLogCursor{file_id=1, log_id=8636, offset=2450786}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947dc050, len:3015, log_data_len:297, seq:8635, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:11.954082] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=89] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8636, offset=2450786}, end_cursor:ObLogCursor{file_id=1, log_id=8637, offset=2451083}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947e0050, len:2718, log_data_len:297, seq:8636, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:11.974390] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:11.995540] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A0-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:11.995585] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A0-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326271511167004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326271511167004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:11.998432] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1A79C2A0-0-0] [lt=97] handle gts request statistics(total_rt=94709, total_cnt=307, avg_rt=3.074967532467532578e+02, avg_cnt=3.069999999999999929e+01) [2025-01-08 00:51:12.021104] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58452][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:12.021982] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread created(tenant_->id()=1002, group_id_=1, token=5) [2025-01-08 00:51:12.024665] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.080252] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=29] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:12.080300] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=36] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=104) [2025-01-08 00:51:12.104444] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=45] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:12.117296] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=71] ts response task statistics(alloc_count=59906, free_count=29952) [2025-01-08 00:51:12.119527] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A457D-0-0] [lt=45] init point thread id with(&point=0x5572d50e3e40, idx_=1108, point=[thread id=58452, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58452) [2025-01-08 00:51:12.126900] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:12.127274] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.137925] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=45] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:12.138566] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18670600}], scn_array=[cnt:1, {val:1736326272117597000, v:0}], curr_size=122, accum_size=911, time_cost=11129) [2025-01-08 00:51:12.138816] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2165000}], scn_array=[cnt:1, {val:1736326272117597000, v:0}], curr_size=122, accum_size=976, time_cost=11123) [2025-01-08 00:51:12.139990] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=41] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=692054, clean_num=31457) [2025-01-08 00:51:12.151955] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=86] new mysql sessid created(conn.sessid_=3221582266, support_ssl=false) [2025-01-08 00:51:12.152022] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] sm conn init succ(conn.sessid_=3221582266, sess.client_addr_="127.0.0.1:51634") [2025-01-08 00:51:12.152040] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2ad8a0dfc030, session_id:3221582266, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:156}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.152072] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221582267, support_ssl=false) [2025-01-08 00:51:12.152103] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221582267, sess.client_addr_="127.0.0.1:51638") [2025-01-08 00:51:12.152115] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8a0de8db0, session_id:3221582267, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.152140] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221582268, support_ssl=false) [2025-01-08 00:51:12.152215] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221582268, sess.client_addr_="127.0.0.1:51640") [2025-01-08 00:51:12.152232] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x2ad8a0dea030, session_id:3221582268, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:193}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.152258] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221582269, support_ssl=false) [2025-01-08 00:51:12.152291] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221582269, sess.client_addr_="127.0.0.1:51642") [2025-01-08 00:51:12.152304] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0deadb0, session_id:3221582269, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:194}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.152327] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221582270, support_ssl=false) [2025-01-08 00:51:12.152356] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221582270, sess.client_addr_="127.0.0.1:51648") [2025-01-08 00:51:12.152367] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0dec030, session_id:3221582270, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:195}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.152390] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221582272, support_ssl=false) [2025-01-08 00:51:12.152417] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221582272, sess.client_addr_="127.0.0.1:51650") [2025-01-08 00:51:12.152429] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8a0decdb0, session_id:3221582272, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:196}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.152451] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221582271, support_ssl=false) [2025-01-08 00:51:12.152479] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221582271, sess.client_addr_="127.0.0.1:51652") [2025-01-08 00:51:12.152490] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8a0dee030, session_id:3221582271, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:197}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.152511] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221582273, support_ssl=false) [2025-01-08 00:51:12.152539] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221582273, sess.client_addr_="127.0.0.1:51654") [2025-01-08 00:51:12.152550] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0deedb0, session_id:3221582273, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:198}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.152619] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=103] new mysql sessid created(conn.sessid_=3221582274, support_ssl=false) [2025-01-08 00:51:12.152663] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221582274, sess.client_addr_="127.0.0.1:51630") [2025-01-08 00:51:12.152676] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8efff8030, session_id:3221582274, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:200}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.156469] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.156520] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=24] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.156551] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.156603] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=31] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.156623] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.156616] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185798-0-0] [lt=190][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193777070448, cache_obj->added_lc()=false, cache_obj->get_object_id()=13610, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:12.156644] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.156659] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.156731] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.156751] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.156773] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.156787] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.156803] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.156817] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.156833] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.156846] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.156864] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.156878] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.156893] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.156918] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=218] new mysql sessid created(conn.sessid_=3221582275, support_ssl=false) [2025-01-08 00:51:12.156995] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221582275, sess.client_addr_="127.0.0.1:51632") [2025-01-08 00:51:12.157013] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x2ad8efff8db0, session_id:3221582275, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:201}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.157042] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221582276, support_ssl=false) [2025-01-08 00:51:12.157106] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221582276, sess.client_addr_="127.0.0.1:51636") [2025-01-08 00:51:12.157059] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD1-0-0] [lt=120] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12276, add_timestamp:1736326272157033}) [2025-01-08 00:51:12.157122] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x2ad8efffadb0, session_id:3221582276, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:202}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.157148] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221582277, support_ssl=false) [2025-01-08 00:51:12.157250] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221582277, sess.client_addr_="127.0.0.1:51644") [2025-01-08 00:51:12.157268] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2ad8efffc030, session_id:3221582277, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:203}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.157294] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221582278, support_ssl=false) [2025-01-08 00:51:12.157407] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221582278, sess.client_addr_="127.0.0.1:51646") [2025-01-08 00:51:12.157424] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x2ad8efffcdb0, session_id:3221582278, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:204}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.157693] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596B-0-0] [lt=18] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12276, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:12.157854] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84290731}], scn_array=[cnt:1, {val:1736326272106664000, v:0}], curr_size=122, accum_size=1961, time_cost=16787) [2025-01-08 00:51:12.157906] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5F-0-0] [lt=52] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.158921] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A5F-0-0] [lt=23] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582274, proxy_sessid=0, client_sessid=3221582274, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.159060] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A60-0-0] [lt=30] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.159894] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A60-0-0] [lt=14] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582273, proxy_sessid=0, client_sessid=3221582273, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.160039] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A61-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.160857] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A61-0-0] [lt=14] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582271, proxy_sessid=0, client_sessid=3221582271, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.160942] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A62-0-0] [lt=22] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.161735] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A62-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582272, proxy_sessid=0, client_sessid=3221582272, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.161814] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A63-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.162575] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A63-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582270, proxy_sessid=0, client_sessid=3221582270, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.162657] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A64-0-0] [lt=24] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.162725] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596B-0-0] [lt=104] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12276, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272162703, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272162703, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272162703, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.162767] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596B-0-0] [lt=40] success to process renew task(task={tenant_id:1, table_id:12276, add_timestamp:1736326272157033}, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272162703, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272162703, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272162703, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.163514] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A64-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582269, proxy_sessid=0, client_sessid=3221582269, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.163596] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A65-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.164316] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A65-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582268, proxy_sessid=0, client_sessid=3221582268, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.164399] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A66-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.165105] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A66-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582267, proxy_sessid=0, client_sessid=3221582267, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.165187] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A67-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.165936] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A67-0-0] [lt=85] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582266, proxy_sessid=0, client_sessid=3221582266, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.167511] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B81-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10008, add_timestamp:1736326272167479}) [2025-01-08 00:51:12.167646] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B83-0-0] [lt=19] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11003, add_timestamp:1736326272167642}) [2025-01-08 00:51:12.167995] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18579A-0-0] [lt=21][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:12.168228] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18579B-0-0] [lt=14] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11008, add_timestamp:1736326272168224}) [2025-01-08 00:51:12.172499] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B81-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.172856] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.172934] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=39] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.172959] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.172997] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.173019] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.173053] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.173075] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:12.173094] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:12.173139] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596C-0-0] [lt=25] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=10008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:12.174664] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7A-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.175737] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7A-0-0] [lt=20] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582278, proxy_sessid=0, client_sessid=3221582278, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.175833] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7B-0-0] [lt=29] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.176701] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7B-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582277, proxy_sessid=0, client_sessid=3221582277, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.176787] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7C-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.177514] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7C-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582276, proxy_sessid=0, client_sessid=3221582276, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.177594] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7D-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:12.178339] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7D-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221582275, proxy_sessid=0, client_sessid=3221582275, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:12.180550] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B85-0-0] [lt=74][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.181677] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46387][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B86-0-0] [lt=20][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.181779] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46387][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B86-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.182012] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B87-0-0] [lt=23] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:12.182060] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46497][T1_L0_G0][T1001][YB42C0A89F89-00062B2D1B484B87-0-0] [lt=16] get_stable_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:12.182088] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46497][T1_L0_G0][T1002][YB42C0A89F89-00062B2D1B484B87-0-0] [lt=10] get_stable_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:12.184118] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B82-0-0] [lt=28] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10008, add_timestamp:1736326272167485}) [2025-01-08 00:51:12.186163] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B87-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.187521] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B82-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.189442] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43188939}], scn_array=[cnt:1, {val:1736326272167339000, v:0}], curr_size=589, accum_size=5697, time_cost=14567) [2025-01-08 00:51:12.189517] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8030, session_id:3221582274, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.189583] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=60] kill and revert session(conn.sessid_=3221582274, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.189626] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sock ref clean, do destroy(*s={this:0x2ad8efff8030, session_id:3221582274, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.189670] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] unlock session of tenant(conn.sessid_=3221582274, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.189683] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221582274, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B18579B-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:133, rec_:133, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:31, seq_:37}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:38}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.189708] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD3-0-0] [lt=40] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11021, add_timestamp:1736326272189704}) [2025-01-08 00:51:12.190990] INFO [SERVER] run (obmp_disconnect.cpp:83) [46500][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=11] free session successfully(sessid=3221582274) [2025-01-08 00:51:12.191015] INFO [SERVER] run (obmp_disconnect.cpp:90) [46500][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=25] mark session id unused(sessid=3221582274) [2025-01-08 00:51:12.191138] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD4-0-0] [lt=7] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12217, add_timestamp:1736326272191134}) [2025-01-08 00:51:12.191164] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD4-0-0] [lt=16] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12142, add_timestamp:1736326272191163}) [2025-01-08 00:51:12.191260] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD4-0-0] [lt=80] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11035, add_timestamp:1736326272191258}) [2025-01-08 00:51:12.191277] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD4-0-0] [lt=10] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12072, add_timestamp:1736326272191276}) [2025-01-08 00:51:12.191291] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD4-0-0] [lt=8] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12068, add_timestamp:1736326272191290}) [2025-01-08 00:51:12.196820] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_stat.cpp:301) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD3-0-0] [lt=21][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:12.197270] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD5-0-0] [lt=26] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11017, add_timestamp:1736326272197265}) [2025-01-08 00:51:12.200534] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46386][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD6-0-0] [lt=42][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326272200529}) [2025-01-08 00:51:12.200561] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46386][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD6-0-0] [lt=27][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326272200529}) [2025-01-08 00:51:12.200572] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46386][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD6-0-0] [lt=10][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=11017, vtable_locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266289, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266289, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326257266289, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.201836] WDIAG get_refactored (ob_hashtable.h:1120) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18579D-0-0] [lt=18][errcode=-4201] hashtable not init [2025-01-08 00:51:12.203371] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8db0, session_id:3221582275, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326272187764, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.203417] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38] kill and revert session(conn.sessid_=3221582275, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.203429] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] sock ref clean, do destroy(*s={this:0x2ad8efff8db0, session_id:3221582275, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326272187764, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.203468] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221582275, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.203480] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221582275, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CD3-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:688, rec_:688, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:219, seq_:225}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:226}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.203619] INFO [SERVER] run (obmp_disconnect.cpp:83) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=17] free session successfully(sessid=3221582275) [2025-01-08 00:51:12.203641] INFO [SERVER] run (obmp_disconnect.cpp:90) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=21] mark session id unused(sessid=3221582275) [2025-01-08 00:51:12.206970] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46386][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD6-0-0] [lt=32][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:12.209358] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=90][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84290853}], scn_array=[cnt:1, {val:1736326272167339000, v:0}], curr_size=586, accum_size=2547, time_cost=22839) [2025-01-08 00:51:12.210075] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0de8db0, session_id:3221582267, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.210110] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221582267, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.210120] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0deadb0, session_id:3221582269, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.210137] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221582269, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.210143] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dfc030, session_id:3221582266, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.210159] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221582266, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.210230] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8a0de8db0, session_id:3221582267, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.210338] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] unlock session of tenant(conn.sessid_=3221582267, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.210354] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221582267, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B81-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.210458] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=100] sock ref clean, do destroy(*s={this:0x2ad8a0deadb0, session_id:3221582269, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.210565] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] unlock session of tenant(conn.sessid_=3221582269, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.210579] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221582269, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B83-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:98, rec_:98, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:10}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:11}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.210609] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] sock ref clean, do destroy(*s={this:0x2ad8a0dfc030, session_id:3221582266, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.210642] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221582266, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.210650] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221582266, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B80-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:134, rec_:134, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:7}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.210815] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=11] free session successfully(sessid=3221582267) [2025-01-08 00:51:12.210829] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=14] mark session id unused(sessid=3221582267) [2025-01-08 00:51:12.210873] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221582269) [2025-01-08 00:51:12.210882] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221582269) [2025-01-08 00:51:12.210926] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221582266) [2025-01-08 00:51:12.210936] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221582266) [2025-01-08 00:51:12.214367] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596C-0-0] [lt=18] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10008, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272214342, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272214342, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272214342, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.214467] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596C-0-0] [lt=98] success to process renew task(task={tenant_id:1, table_id:10008, add_timestamp:1736326272167479}, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272214342, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272214342, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272214342, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.214513] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596D-0-0] [lt=28] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11003, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:12.214793] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=107] ====== traversal_flush timer task ====== [2025-01-08 00:51:12.214953] INFO [MDS] flush (mds_table_mgr.cpp:195) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48536D-0-0] [lt=20] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326271767610002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326280087570000, latest_log_ts:{val:1736326263438968001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.215078] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48536D-0-0] [lt=92] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326267132611000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.215161] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48536D-0-0] [lt=55] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326267147675000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.215753] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48536D-0-0] [lt=554] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326267189859000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.215835] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48536D-0-0] [lt=79] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:12.220762] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dea030, session_id:3221582268, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.220868] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=98] kill and revert session(conn.sessid_=3221582268, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.220879] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dee030, session_id:3221582271, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326272167339, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.220896] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221582271, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.220877] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffcdb0, session_id:3221582278, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326272187764, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.220903] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0decdb0, session_id:3221582272, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326272167339, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.220918] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221582272, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.220917] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=33] kill and revert session(conn.sessid_=3221582278, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.220928] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] sock ref clean, do destroy(*s={this:0x2ad8efffcdb0, session_id:3221582278, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326272187764, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.220966] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221582278, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.220977] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221582278, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CD6-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:549, rec_:549, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:12, seq_:19}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:20}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.220996] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8a0dea030, session_id:3221582268, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326272166080, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.221090] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] unlock session of tenant(conn.sessid_=3221582268, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.221103] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221582268, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B82-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:436, rec_:436, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.221239] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=132] sock ref clean, do destroy(*s={this:0x2ad8a0dee030, session_id:3221582271, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326272167339, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.221283] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] free session successfully(sessid=3221582278) [2025-01-08 00:51:12.221328] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=44] mark session id unused(sessid=3221582278) [2025-01-08 00:51:12.221311] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221582271, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.221381] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=7] free session successfully(sessid=3221582268) [2025-01-08 00:51:12.221365] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=53] connection close(sessid=3221582271, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B86-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.221391] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221582268) [2025-01-08 00:51:12.221396] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] sock ref clean, do destroy(*s={this:0x2ad8a0decdb0, session_id:3221582272, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326272167339, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.221443] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221582272, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.221454] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221582272, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B85-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:633, rec_:633, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.221487] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dec030, session_id:3221582270, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326272167339, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.221500] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221582271) [2025-01-08 00:51:12.221506] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] kill and revert session(conn.sessid_=3221582270, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.221512] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=12] mark session id unused(sessid=3221582271) [2025-01-08 00:51:12.221563] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221582272) [2025-01-08 00:51:12.221573] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221582272) [2025-01-08 00:51:12.221575] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8a0dec030, session_id:3221582270, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326272167339, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.221624] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] unlock session of tenant(conn.sessid_=3221582270, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.221661] INFO [SERVER] run (obmp_disconnect.cpp:83) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=7] free session successfully(sessid=3221582270) [2025-01-08 00:51:12.221680] INFO [SERVER] run (obmp_disconnect.cpp:90) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=19] mark session id unused(sessid=3221582270) [2025-01-08 00:51:12.221635] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221582270, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B84-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:368, rec_:368, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.227423] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.228267] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596D-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11003, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272228237, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272228237, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272228237, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.228315] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596D-0-0] [lt=47] success to process renew task(task={tenant_id:1, table_id:11003, add_timestamp:1736326272167642}, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272228237, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272228237, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272228237, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.228359] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596E-0-0] [lt=27] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:12.231814] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2165122}], scn_array=[cnt:1, {val:1736326272117597001, v:0}], curr_size=122, accum_size=1098, time_cost=12852) [2025-01-08 00:51:12.235387] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596E-0-0] [lt=18] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11008, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235366, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235366, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235366, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.235432] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596E-0-0] [lt=44] success to process renew task(task={tenant_id:1, table_id:11008, add_timestamp:1736326272168224}, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235366, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235366, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235366, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.235469] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596F-0-0] [lt=26] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=10008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:12.235965] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596F-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10008, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235952, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235952, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235952, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.236003] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28596F-0-0] [lt=37] success to process renew task(task={tenant_id:1, table_id:10008, add_timestamp:1736326272167485}, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235952, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235952, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272235952, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.236035] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285970-0-0] [lt=24] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11021, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:12.237181] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285970-0-0] [lt=9] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11021, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237168, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237168, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237168, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.237395] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285970-0-0] [lt=213] success to process renew task(task={tenant_id:1, table_id:11021, add_timestamp:1736326272189704}, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237168, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237168, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237168, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.237495] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285971-0-0] [lt=31] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12217, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237482, sql_renew_time:1736326272237482, is_mark_fail:false}]) [2025-01-08 00:51:12.237517] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285971-0-0] [lt=22] success to process renew task(task={tenant_id:1, table_id:12217, add_timestamp:1736326272191134}, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237482, sql_renew_time:1736326272237482, is_mark_fail:false}]) [2025-01-08 00:51:12.237543] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285972-0-0] [lt=14] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12142, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237537, sql_renew_time:1736326272237537, is_mark_fail:false}]) [2025-01-08 00:51:12.237557] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285972-0-0] [lt=14] success to process renew task(task={tenant_id:1, table_id:12142, add_timestamp:1736326272191163}, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237537, sql_renew_time:1736326272237537, is_mark_fail:false}]) [2025-01-08 00:51:12.237577] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285973-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11035, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237573, sql_renew_time:1736326272237573, is_mark_fail:false}]) [2025-01-08 00:51:12.237591] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285973-0-0] [lt=13] success to process renew task(task={tenant_id:1, table_id:11035, add_timestamp:1736326272191258}, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237573, sql_renew_time:1736326272237573, is_mark_fail:false}]) [2025-01-08 00:51:12.237610] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285974-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12072, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237606, sql_renew_time:1736326272237606, is_mark_fail:false}]) [2025-01-08 00:51:12.237690] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285974-0-0] [lt=79] success to process renew task(task={tenant_id:1, table_id:12072, add_timestamp:1736326272191276}, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237606, sql_renew_time:1736326272237606, is_mark_fail:false}]) [2025-01-08 00:51:12.237721] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285975-0-0] [lt=14] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12068, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237712, sql_renew_time:1736326272237712, is_mark_fail:false}]) [2025-01-08 00:51:12.237736] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285975-0-0] [lt=14] success to process renew task(task={tenant_id:1, table_id:12068, add_timestamp:1736326272191290}, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272237712, sql_renew_time:1736326272237712, is_mark_fail:false}]) [2025-01-08 00:51:12.237754] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285976-0-0] [lt=12] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11017, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:12.238483] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285976-0-0] [lt=9] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11017, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272238472, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272238472, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272238472, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.238519] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285976-0-0] [lt=35] success to process renew task(task={tenant_id:1, table_id:11017, add_timestamp:1736326272197265}, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272238472, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272238472, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326272238472, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:12.243938] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD5-0-0] [lt=73][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:12.245477] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=83][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0deedb0, session_id:3221582273, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326272167339, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.245521] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221582273, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.245532] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] sock ref clean, do destroy(*s={this:0x2ad8a0deedb0, session_id:3221582273, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326272167339, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.245568] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221582273, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.245579] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221582273, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B87-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:562, rec_:562, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.245727] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221582273) [2025-01-08 00:51:12.245742] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=15] mark session id unused(sessid=3221582273) [2025-01-08 00:51:12.245840] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=108][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43189528}], scn_array=[cnt:1, {val:1736326272220868000, v:0}], curr_size=589, accum_size=6286, time_cost=20466) [2025-01-08 00:51:12.246007] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2198636}], scn_array=[cnt:1, {val:1736326272010447002, v:0}], curr_size=122, accum_size=1210, time_cost=20683) [2025-01-08 00:51:12.257344] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:12.257906] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffadb0, session_id:3221582276, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326272187764, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.257966] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=49] kill and revert session(conn.sessid_=3221582276, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.257984] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffc030, session_id:3221582277, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326272187764, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.258013] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221582277, proxy_sessid=0, ret=0) [2025-01-08 00:51:12.258026] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] sock ref clean, do destroy(*s={this:0x2ad8efffadb0, session_id:3221582276, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326272187764, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.258072] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] unlock session of tenant(conn.sessid_=3221582276, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.258089] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221582276, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CD4-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:119, rec_:119, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:2, seq_:4}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:5}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.258147] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=54] sock ref clean, do destroy(*s={this:0x2ad8efffc030, session_id:3221582277, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326272187764, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:12.258333] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] unlock session of tenant(conn.sessid_=3221582277, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:12.258351] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] connection close(sessid=3221582277, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CD5-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:241, rec_:241, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:12.258469] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221582276) [2025-01-08 00:51:12.258494] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=24] mark session id unused(sessid=3221582276) [2025-01-08 00:51:12.258543] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] free session successfully(sessid=3221582277) [2025-01-08 00:51:12.258554] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221582277) [2025-01-08 00:51:12.261603] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1003}, {id:1001}]) [2025-01-08 00:51:12.265891] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=34] gc stale ls task succ [2025-01-08 00:51:12.266074] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] [COMPACTION DAG STATUS] (start_time=1736326261869461, end_time=1736326272266072, dag_status={}) [2025-01-08 00:51:12.266302] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=222] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:12.266320] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:12.266328] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:12.266336] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:12.266343] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:12.266350] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:12.266358] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:12.266365] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:12.266371] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:12.266379] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266390] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266398] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266406] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266414] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266422] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266430] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266438] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266446] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266455] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266462] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266471] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266479] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266487] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266495] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266502] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266510] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266518] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266526] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266534] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266542] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266550] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266558] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266566] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266573] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266581] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266589] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266597] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266605] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266613] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266621] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266629] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266637] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266644] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266652] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266660] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266668] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266675] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266684] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266691] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266699] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266708] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266716] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266723] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266732] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266740] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266747] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266755] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266763] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266778] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266786] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266794] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266802] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266810] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266818] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266825] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266833] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266841] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266849] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266857] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266864] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266872] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266880] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266887] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266895] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266903] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266911] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266919] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266927] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266935] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266943] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266951] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:12.266960] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:12.267009] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=48] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:12.267021] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:12.267027] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:12.267032] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:12.267037] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:12.267042] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:12.267047] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:12.267052] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:12.267097] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=44] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:12.267105] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:12.267146] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=67] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:12.283982] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3A7-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.284424] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=52] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:12.287654] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:12.289839] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=16] ls blacklist refresh finish(cost_time=36176) [2025-01-08 00:51:12.295852] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:12.298616] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD7-0-0] [lt=25] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:12.298667] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD7-0-0] [lt=42] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=53) [2025-01-08 00:51:12.306678] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:12.311516] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:51:12.313807] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:12.313842] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] temporary file module use no blocks [2025-01-08 00:51:12.313853] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:12.313867] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:12.313880] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193776227472, flush_io_finished_ret_:0, flush_io_finished_round_:1935, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:12.314686] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=45] ====== checkpoint timer task ====== [2025-01-08 00:51:12.315562] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:12.315605] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=42] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:12.315676] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=24] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326280087570000, latest_log_ts:{val:1736326263438968001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.315757] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=78] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:12.315776] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:12.315790] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326263438968001, v:0}) [2025-01-08 00:51:12.315798] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:12.315812] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:12.315830] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:12.315903] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=69] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:12.315927] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:12.315948] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:12.316768] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:12.316799] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=31] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:12.316837] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=16] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326267132611000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.316895] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=57] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:12.316907] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:12.316919] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:12.316933] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:12.316947] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2165244}, scn={val:1736326272117597001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:12.317084] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:12.317800] INFO [PALF] pread (log_reader.cpp:130) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1653) [2025-01-08 00:51:12.319468] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=39] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8637, offset=2451083}, end_cursor:ObLogCursor{file_id=1, log_id=8638, offset=2451380}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947e4050, len:2421, log_data_len:297, seq:8637, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:12.319576] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326272117597001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:12.319607] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:12.319804] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=195] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:12.321018] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:12.321056] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=38] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:12.321100] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326267147675000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.321366] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=263] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:12.321380] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:12.321395] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:12.321410] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:12.321424] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2198636}, scn={val:1736326272010447001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:12.321557] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.321708] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=110] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:12.321677] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:12.321940] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=106][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:12.322025] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.323397] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=57] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:12.324772] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8638, offset=2451380}, end_cursor:ObLogCursor{file_id=1, log_id=8639, offset=2451677}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947e8050, len:2124, log_data_len:297, seq:8638, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:12.324851] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326272010447001, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:12.324879] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:12.324899] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:12.324913] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=8] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:12.324932] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=14] get_min_unreplayed_log_info(lsn={lsn:2186359}, scn={val:1736326272264082001, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326272307816, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2186359}, next_to_submit_scn_:{val:1736326272264082001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2186237}, end_lsn:{lsn:2186359}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2186359}, readable_log_tail:{lsn:2186359}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64918409, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326272264082000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326272264082000, v:0}, data_checksum:2124447694, flag:19978}}, init_mode_version:0, accumulate_checksum:791920596, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326272264082000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17872, miss_cnt:0, cache_read_size:2186359, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:12.326747] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=150] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:12.327240] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=492] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:12.327400] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326267189859000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.327607] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=205] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:12.327622] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:12.327637] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:12.327651] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:12.327665] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2186359}, scn={val:1736326272264082000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:12.328258] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:12.328320] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.330281] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=39] success to update min reserved snapshot(reserved_snapshot=1736324465706032000, duration=1800, snapshot_gc_ts=1736326265706032000) [2025-01-08 00:51:12.332819] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=27] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8639, offset=2451677}, end_cursor:ObLogCursor{file_id=1, log_id=8640, offset=2451974}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947f0050, len:1827, log_data_len:297, seq:8639, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:12.332950] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=16] update reserved snapshot finished(cost_ts=2670, reserved_snapshot=1736324465706032000) [2025-01-08 00:51:12.333089] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1736326272333086, bytes: 872961498, bw: 0.228395 MB/s, add_ts: 1002782, add_bytes: 240156 [2025-01-08 00:51:12.333487] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326272264082000, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:12.333520] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:12.333541] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:12.333551] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:12.340447] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAE-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:12.340479] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAE-0-0] [lt=28] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:12.340711] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:12.358572] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:12.358603] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:12.361596] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87085-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.361639] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:12.361651] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:12.364825] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=37] current monitor number(seq_=-1) [2025-01-08 00:51:12.367900] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185423-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.370947] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:12.387018] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58436][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA683C-0-0] [lt=85] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4783) [2025-01-08 00:51:12.391916] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=67] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:12.400289] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=84] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:12.404774] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58404][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2A-0-0] [lt=17] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326272407785, keepalive_ts:-1}) [2025-01-08 00:51:12.404829] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58404][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2A-0-0] [lt=53] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326272407785, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:12.419508] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D040-0-0] [lt=26] need check schema(new_schema_version=1736326191428712, cached_tenant_schema_version=1736326191428712, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = ? ORDER BY row_id, column_name) [2025-01-08 00:51:12.419822] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4583-0-0] [lt=102] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:51:12.419938] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4583-0-0] [lt=45] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:12.419956] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4583-0-0] [lt=17] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=45447, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:12.419964] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4583-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=32, avg_log_submit_to_flush_time=4744, avg_log_submit_to_slide_time=45415) [2025-01-08 00:51:12.420108] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:12.420145] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:12.420163] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:12.420400] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=232] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:12.420713] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:12.420752] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17973, miss_cnt:0, cache_read_size:2198880, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:12.421424] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=24] ====== traversal_flush timer task ====== [2025-01-08 00:51:12.421920] INFO [MDS] flush (mds_table_mgr.cpp:195) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E52-0-0] [lt=88] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326272220868000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326279284252000, latest_log_ts:{val:1736326262663505001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.422093] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E52-0-0] [lt=167] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:12.422496] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4232, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:12.422533] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:12.422561] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=986, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:12.422728] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=33, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:12.422847] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=109] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1643) [2025-01-08 00:51:12.424713] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:12.425104] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8555-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=25, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:12.425125] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8555-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7072, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:12.425484] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=77] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:12.428031] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D041-0-0] [lt=149][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.428709] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.428752] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=4509, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:12.428769] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1454) [2025-01-08 00:51:12.428786] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=487, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:12.428833] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=42, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:12.433384] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:12.446014] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=27] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:12.446075] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=57] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:12.446091] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:12.446102] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:12.448690] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=132, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}]) [2025-01-08 00:51:12.449257] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=564] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:12.459334] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=108] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.459457] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.477279] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58436][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F3-0-0] [lt=54] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326272480152, keepalive_ts:-1}) [2025-01-08 00:51:12.477302] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58436][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F3-0-0] [lt=22] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326272480152, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:12.482347] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B2-0-0] [lt=7] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326272484741, keepalive_ts:-1}) [2025-01-08 00:51:12.482392] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B2-0-0] [lt=44] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326272484741, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:12.483603] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58452][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B3-0-0] [lt=42] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326272485003, keepalive_ts:-1}) [2025-01-08 00:51:12.483634] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58452][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B3-0-0] [lt=29] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326272485003, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:12.509099] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=344, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:12.509130] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=7, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:12.519439] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=46] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.520068] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4585-0-0] [lt=16][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18066, log_task={header:{begin_lsn:{lsn:2198880}, end_lsn:{lsn:2199002}, log_id:18066, min_scn:{val:1736326272010447004, v:0}, max_scn:{val:1736326272010447004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2198758}, prev_proposal_id:1, committed_end_lsn:{lsn:2198880}, data_checksum:1715074492, accum_checksum:4206169455, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326272419787, freeze_ts:1736326272419787, submit_ts:1736326272419798, flushed_ts:1736326272428804, gen_to_freeze cost time:0, gen_to_submit cost time:11, submit_to_flush cost time:9006}, fs_cb_begin_ts=1736326272520045, log_life_time=100258) [2025-01-08 00:51:12.520428] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:12.520450] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] try gc retain ctx [2025-01-08 00:51:12.520458] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:12.520490] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326272117597003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326261935253004, v:0}, keep_alive_scn_:{val:1736326267132611003, v:0}, update_ts_:1736326272519941}) [2025-01-08 00:51:12.521402] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326280087570000, latest_log_ts:{val:1736326263438968001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:12.521634] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=84] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:12.521654] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:12.521682] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] finish update min start scn(max_decided_scn={val:1736326272117597003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326261935253004, v:0}, keep_alive_scn_:{val:1736326267132611003, v:0}, update_ts_:1736326272519941}) [2025-01-08 00:51:12.522414] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326272117597001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:12.522552] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] finish update min start scn(max_decided_scn={val:1736326272010447003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326264665540004, v:0}, keep_alive_scn_:{val:1736326270015178001, v:0}, update_ts_:1736326272519941}) [2025-01-08 00:51:12.523283] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326272010447001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:12.523352] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=58] finish update min start scn(max_decided_scn={val:1736326272264082002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326263623509000, v:0}, keep_alive_scn_:{val:1736326267890517000, v:0}, update_ts_:1736326272519941}) [2025-01-08 00:51:12.524393] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326272264082000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326267890517005, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:12.528440] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:12.529010] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:12.529140] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=127][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.541443] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:12.542311] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.544176] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.545661] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4197) [2025-01-08 00:51:12.545703] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:12.554816] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58459][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:12.556130] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:51:12.557399] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:12.557458] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:52445, [delay/us]:prepare:7, schedule:80, submit:770, rt:3838, total:4696) [2025-01-08 00:51:12.557475] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:12.557577] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52446, [delay/us]: prepare:7, schedule:80, submit:770, device:3838, total:4696) [2025-01-08 00:51:12.557594] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:12.557636] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:18, bw:80686, [delay/us]:prepare:7, schedule:430, submit:126, rt:2886, total:3451) [2025-01-08 00:51:12.557647] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:12.557686] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:18, avg_bw:80686, [delay/us]: prepare:7, schedule:430, submit:126, device:2886, total:3451) [2025-01-08 00:51:12.557696] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:12.557718] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:7, bw:28240, [delay/us]:prepare:4, schedule:156, submit:1, rt:1510, total:1673) [2025-01-08 00:51:12.557741] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4198, iops:39, bw:165408, [delay/us]:prepare:10, schedule:447, submit:200, rt:2613, total:3270) [2025-01-08 00:51:12.557800] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12103, [delay/us]:prepare:19, schedule:1224, submit:150, rt:329, total:1723) [2025-01-08 00:51:12.557815] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:12.557847] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:7, avg_bw:28240, [delay/us]: prepare:4, schedule:156, submit:1, device:1510, total:1673) [2025-01-08 00:51:12.557862] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12103, [delay/us]: prepare:19, schedule:1224, submit:150, device:329, total:1723) [2025-01-08 00:51:12.557870] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4198, avg_iops:40, avg_bw:165406, [delay/us]: prepare:10, schedule:447, submit:200, device:2613, total:3270) [2025-01-08 00:51:12.557880] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:12.557900] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4034, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:458, total:466) [2025-01-08 00:51:12.557924] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:12.557950] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4034, [delay/us]: prepare:0, schedule:0, submit:600000000, device:458, total:466) [2025-01-08 00:51:12.557968] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:12.557984] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:12.557992] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:12.558001] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:12.558008] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=240kB/s, net_out=249kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:12.558876] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15008396051, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:12.558978] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:12.558995] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try_wait return eagain(this=0x2ad8881fc3b0, thread=0x2ad8aedf1960) [2025-01-08 00:51:12.559003] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15008396051, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:12.559014] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7771593730, tenant_id:1001}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:12.559025] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:12.559031] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6] try_wait return eagain(this=0x2ad8f90b0e70, thread=0x2ad9047efe50) [2025-01-08 00:51:12.559036] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=5] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7771593730, tenant_id:1001}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:12.588279] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C97-0-0] [lt=53][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.588340] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [58428][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=21] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:12.593170] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C97-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:12.601940] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4E-0-0] [lt=167] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326272695516, keepalive_ts:-1}) [2025-01-08 00:51:12.601978] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58379][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4E-0-0] [lt=37] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326272695516, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:12.612077] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C62C-0-0] [lt=24][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193777525909, cache_obj->added_lc()=false, cache_obj->get_object_id()=13611, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:12.613316] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3829-0-0] [lt=16] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:12.613366] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3829-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:12.613385] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3829-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:12.618149] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=59] submit log success(service_type_=0, last_id=1736326289254996000, limited_id=1736326289254996000) [2025-01-08 00:51:12.618189] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=39] submit log with lock success(service_type_=0, last_id=1736326289254996000, limited_id=1736326289254996000) [2025-01-08 00:51:12.618279] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326272618044], gts:1736326272617414000, latest_srr:[mts=1736326272618044]}) [2025-01-08 00:51:12.618299] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:12.618386] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326272618310], gts:1736326272617414000, latest_srr:[mts=1736326272618310]}) [2025-01-08 00:51:12.618854] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326272618839], gts:1736326272617414000, latest_srr:[mts=1736326272618839]}) [2025-01-08 00:51:12.621893] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4587-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:12.621996] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4587-0-0] [lt=82][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18067, log_task={header:{begin_lsn:{lsn:2199002}, end_lsn:{lsn:2199124}, log_id:18067, min_scn:{val:1736326272511614000, v:0}, max_scn:{val:1736326272511614000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2198880}, prev_proposal_id:1, committed_end_lsn:{lsn:2199002}, data_checksum:1715074492, accum_checksum:463390557, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326272519997, freeze_ts:1736326272519997, submit_ts:1736326272520006, flushed_ts:1736326272526966, gen_to_freeze cost time:0, gen_to_submit cost time:9, submit_to_flush cost time:6960}, fs_cb_begin_ts=1736326272621983, log_life_time=101986) [2025-01-08 00:51:12.623446] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1736326272420764000, v:0}, base_version={val:1736326272220868000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326272420764000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326272623426, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326272420764000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326272646201, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326272420764000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326272585927, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:12.623803] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=21] update id meta(ret=0, service_type=0, limited_id=1736326289254996000, latest_scn={val:1736326272617414001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289254996000, latest_log_ts:{val:1736326272617414001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:12.623863] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=59] handle submit callback(service_type_=0, success=true, used_ts=5715, log_ts={val:1736326272617414001, v:0}, limited_id=1736326289254996000) [2025-01-08 00:51:12.623876] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=12] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:12.623884] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:12.625893] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=108] submit log success(service_type_=0, last_id=1736326289284252000, limited_id=1736326289284252000) [2025-01-08 00:51:12.625909] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] submit log with lock success(service_type_=0, last_id=1736326289284252000, limited_id=1736326289284252000) [2025-01-08 00:51:12.629886] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.631611] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=20] update id meta(ret=0, service_type=0, limited_id=1736326289284252000, latest_scn={val:1736326272625185001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289284252000, latest_log_ts:{val:1736326272625185001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:12.631699] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=88] handle submit callback(service_type_=0, success=true, used_ts=5808, log_ts={val:1736326272625185001, v:0}, limited_id=1736326289284252000) [2025-01-08 00:51:12.631712] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=12] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:12.631721] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:12.632457] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326272632078000, start_id=1736326272632078000, end_id=1736326272632078001) [2025-01-08 00:51:12.645889] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326268537126, last_first_register_time_us=1736324449183670, register_time_us=1736326268537126, curr_time_us=1736326272645885) [2025-01-08 00:51:12.645990] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=62] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326272645954, last_submit_register_req_time_us=1736326272645954, register_parent_reason=1) [2025-01-08 00:51:12.646009] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:12.693146] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84645-0-0] [lt=37] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2025-01-08 00:51:12.695095] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84645-0-0] [lt=34] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2025-01-08 00:51:12.695145] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1010) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84645-0-0] [lt=47] no transfer partition job [2025-01-08 00:51:12.695161] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1040) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84645-0-0] [lt=11] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer=true) [2025-01-08 00:51:12.698282] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:284) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84645-0-0] [lt=10] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-01-08 00:51:12.698433] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:493) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84645-0-0] [lt=133] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2025-01-08 00:51:12.698471] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:157) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84645-0-0] [lt=30] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=3, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1003}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone3", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1736324440080711, last_statistic_bg_stat_time=1736326192603087, last_statistic_schema_version=1736326192037600, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-01-08 00:51:12.699032] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58544A-0-0] [lt=13] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:12.699077] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58544A-0-0] [lt=36] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:12.700949] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:801) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D1728529C-0-0] [lt=27] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, location:{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, location:{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}]) [2025-01-08 00:51:12.706754] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC382B-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=26, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:12.706786] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC382B-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21343, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:12.706830] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC382B-0-0] [lt=41] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=1854, avg_log_submit_to_slide_time=21328) [2025-01-08 00:51:12.706845] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58460][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:12.706997] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1002, group_id_=1, token=6) [2025-01-08 00:51:12.707249] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:12.707309] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17875, miss_cnt:0, cache_read_size:2186725, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:12.721130] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=106] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=80, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:12.721223] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1634, avg_group_log_size=163) [2025-01-08 00:51:12.721720] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:348K/83:375369 disk:348K/83:375369 t1m2:56K/13:70148 t1m2g0:56K/13:70148 t1002m2:204K/51:446755 t1002m2g0:204K/51:446755 t1001m2:88K/19:392591 t1001m2g0:88K/19:392591 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:12.723006] INFO [LIB] stat (utility.h:1210) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4589-0-0] [lt=145] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:12.725065] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3671, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:12.725092] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1634) [2025-01-08 00:51:12.725170] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1044, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:12.725334] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:12.726467] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58460][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC855D-0-0] [lt=22] init point thread id with(&point=0x5572d50e4040, idx_=1116, point=[thread id=58460, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58460) [2025-01-08 00:51:12.727785] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC855E-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:12.727823] INFO [LIB] stat (utility.h:1210) [58379][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC855E-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7887, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:12.728606] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=75] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:12.730544] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.737473] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-01-08 00:51:12.741989] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:12.745851] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=63, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:12.745884] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=4325, avg_group_log_size=332) [2025-01-08 00:51:12.747515] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=4379, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:12.747544] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4325) [2025-01-08 00:51:12.747564] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=704, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:12.747790] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=31, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:12.753247] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58461][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:12.753360] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:51:12.753444] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6840-0-0] [lt=40] init point thread id with(&point=0x5572d50e4080, idx_=1117, point=[thread id=58461, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58461) [2025-01-08 00:51:12.753540] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6840-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:12.753556] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6840-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=7445, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:12.753565] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6840-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=93, avg_log_submit_to_flush_time=5127, avg_log_submit_to_slide_time=7352) [2025-01-08 00:51:12.754426] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4325, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:12.755813] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D3B-0-0] [lt=171] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:12.759648] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=70] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5229278909, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:12.759701] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:12.759716] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad90857ded0) [2025-01-08 00:51:12.759725] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5229278909, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:12.775994] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ACF3-0-0] [lt=83] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.863697242736816406e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=393, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.343607080527527842e+02) [2025-01-08 00:51:12.781743] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=39] PNIO [ratelimit] time: 1736326272781740, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007750, add_bytes: 0 [2025-01-08 00:51:12.785853] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:12.787934] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58409][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=19] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:12.789153] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085153-0-0] [lt=15] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:12.790440] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326272383788001, v:0}, new_max_version={val:1736326272383788001, v:0}, last_min_version={val:1736326272167339002, v:0}, last_max_version={val:1736326272167339002, v:0}, total_time_us=6672, delta_us=406649) [2025-01-08 00:51:12.790704] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=260] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326272383788001, v:0}, min_version={val:1736326272383788001, v:0}, max_version={val:1736326272383788001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:12.793085] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=57] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:12.796290] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=10398720) [2025-01-08 00:51:12.806845] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=31][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:12.806930] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:12.807911] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=34][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:12.807960] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=17][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:12.819011] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:12.819437] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:12.822702] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58434][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1001, workers_.get_size()=2) [2025-01-08 00:51:12.826963] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=74, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:12.831333] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.833173] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58404][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3932-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10314) [2025-01-08 00:51:12.834583] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:12.835401] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:12.835524] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=111] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=92) [2025-01-08 00:51:12.836001] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=137][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18671445}], scn_array=[cnt:1, {val:1736326272617414002, v:0}], curr_size=122, accum_size=122, time_cost=14218) [2025-01-08 00:51:12.835990] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2199368}], scn_array=[cnt:1, {val:1736326272511614003, v:0}], curr_size=122, accum_size=488, time_cost=15381) [2025-01-08 00:51:12.836352] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=99, avg_log_submit_to_flush_time=6050, avg_log_submit_to_slide_time=8843) [2025-01-08 00:51:12.836493] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:12.839650] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=46] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:12.840508] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2165845}], scn_array=[cnt:1, {val:1736326272617414002, v:0}], curr_size=122, accum_size=479, time_cost=18623) [2025-01-08 00:51:12.841133] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=4218, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:12.841160] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43193556}], scn_array=[cnt:1, {val:1736326272824946000, v:0}], curr_size=589, accum_size=10314, time_cost=13789) [2025-01-08 00:51:12.841241] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=80] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10314) [2025-01-08 00:51:12.841284] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=35] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=190, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:12.841585] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=12, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:12.841648] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT LOG LIFE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=7614, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:12.841661] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=134, avg_log_gen_to_submit_time=227, avg_log_submit_to_flush_time=4639, avg_log_submit_to_slide_time=7386) [2025-01-08 00:51:12.841679] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=31, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:12.841694] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10314, hit_cnt=20, read_cnt=20, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:12.846620] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1001, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:17795, max_lsn:2165967, max_scn:{val:1736326272617414002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2165967}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2165845}, last_submit_end_lsn:{lsn:2165967}, last_submit_log_id:17795, last_submit_log_pid:1, max_flushed_lsn:{lsn:2165845}, max_flushed_end_lsn:{lsn:2165967}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2165967}, last_slide_log_id:17795, last_slide_scn:{val:1736326272617414002, v:0}, last_slide_lsn:{lsn:2165845}, last_slide_end_lsn:{lsn:2165967}, last_slide_log_pid:1, last_slide_log_accum_checksum:4193812776, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924f6a530}, [StateMgr]={this:0x2ad924f6e830, self:"192.168.159.137:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:17796, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324447115984, palf_role_change_cb:0x2ad924f752f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924f6bc70}, [ModeMgr]={palf_id:1001, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324447762546, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436756804001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/meta", dir_fd:177, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924f70a38, flashback_version:0}, log_storage:{log_tail:{lsn:2165967}, readable_log_tail:{lsn:2165967}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64938801, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1736324447114985, this:0x2ad924f6ef70}, [Reconfirm]={palf_id:1001, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:1"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324447117136, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924f6e9b0}) [2025-01-08 00:51:12.847036] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=410] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326272840578, last_flushed_end_lsn:{lsn:2165967}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326272827756, last_flushed_end_lsn:{lsn:2165967}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326272839898, last_flushed_end_lsn:{lsn:2165967}}]) [2025-01-08 00:51:12.847119] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=78] current log_task status(palf_id=1001, self="192.168.159.137:2882", log_id=17796, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326272840578, last_flushed_end_lsn:{lsn:2165967}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326272827756, last_flushed_end_lsn:{lsn:2165967}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326272839898, last_flushed_end_lsn:{lsn:2165967}}]) [2025-01-08 00:51:12.854079] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=36] ====== traversal_flush timer task ====== [2025-01-08 00:51:12.854295] INFO [MDS] flush (mds_table_mgr.cpp:195) [58423][T1_TimerWK4][T1][YB42C0A89F89-00062B2D28184593-0-0] [lt=30] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326272783501000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289254996000, latest_log_ts:{val:1736326272617414001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:12.854422] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [58423][T1_TimerWK4][T1][YB42C0A89F89-00062B2D28184593-0-0] [lt=120] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:12.864647] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=10] BlockGCTimerTask success(ret=0, cost_time_us=15, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:12.864808] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F7-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:12.864910] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F7-0-0] [lt=81] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326272761535000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326272761535000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:12.880872] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50623][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F7-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=10314, avg_group_log_size=542) [2025-01-08 00:51:12.880923] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50623][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F7-0-0] [lt=44] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10314) [2025-01-08 00:51:12.885986] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=26] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:12.896301] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.896342] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.896368] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.896390] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:12.911041] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1663, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:12.911161] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=116] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:12.911189] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=429, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:12.911263] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:12.923365] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58402][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A458D-0-0] [lt=18][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18070, log_task={header:{begin_lsn:{lsn:2199368}, end_lsn:{lsn:2199490}, log_id:18070, min_scn:{val:1736326272511614003, v:0}, max_scn:{val:1736326272511614003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2199246}, prev_proposal_id:1, committed_end_lsn:{lsn:2199368}, data_checksum:1715074492, accum_checksum:2825349665, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326272820448, freeze_ts:1736326272820448, submit_ts:1736326272820515, flushed_ts:1736326272836406, gen_to_freeze cost time:0, gen_to_submit cost time:67, submit_to_flush cost time:15891}, fs_cb_begin_ts=1736326272923353, log_life_time=102905) [2025-01-08 00:51:12.928872] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:12.931522] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:12.942662] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6752137216, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:12.960224] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7804043714, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:12.960342] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:12.960360] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] try_wait return eagain(this=0x2ad8c0eb0e70, thread=0x2ad9349e7e40) [2025-01-08 00:51:12.960370] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7804043714, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:12.961391] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8640, offset=2451974}, end_cursor:ObLogCursor{file_id=1, log_id=8641, offset=2452279}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947ec050, len:1530, log_data_len:305, seq:8640, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:12.962599] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4E-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:12.962768] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4E-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326272617414002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326272617414002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:12.973841] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=95] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8641, offset=2452279}, end_cursor:ObLogCursor{file_id=1, log_id=8642, offset=2452576}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947f4050, len:1225, log_data_len:297, seq:8641, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:12.977520] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4E-0-0] [lt=43] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:12.977569] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4E-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326272617414002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326272617414002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:13.004910] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=100] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8642, offset=2452576}, end_cursor:ObLogCursor{file_id=1, log_id=8643, offset=2452873}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947f8050, len:928, log_data_len:297, seq:8642, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:13.011109] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=109] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8643, offset=2452873}, end_cursor:ObLogCursor{file_id=1, log_id=8644, offset=2453170}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a0c050, len:631, log_data_len:297, seq:8643, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:13.022427] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58462][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:13.022649] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58463][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:13.022685] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread created(tenant_->id()=1001, group_id_=1, token=3) [2025-01-08 00:51:13.028877] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A1-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:13.028993] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A1-0-0] [lt=97] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326272511614004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326272511614004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:13.032980] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.033334] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58462][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2C9F-0-0] [lt=44] init point thread id with(&point=0x5572d50e40c0, idx_=1118, point=[thread id=58462, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58462) [2025-01-08 00:51:13.034042] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58464][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:13.034111] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread created(tenant_->id()=1001, group_id_=1, token=4) [2025-01-08 00:51:13.042536] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=163] runTimer to refresh locality_info(ret=0) [2025-01-08 00:51:13.044912] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F8528C-0-0] [lt=11] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326268337893, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:13.044962] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F8528C-0-0] [lt=48] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326268340529, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:13.045036] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F8528C-0-0] [lt=21] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, renew_time:1736326268342974, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, renew_time:1736326268342976, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, renew_time:1736326268342978, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326268342970, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:13.045738] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:346) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=19] set_server_locality_array success(server_locality_array=[{inited:true, addr:"192.168.159.137:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322580423131, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.138:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581270010, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.139:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581175973, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-01-08 00:51:13.045789] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=38] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-01-08 00:51:13.045802] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=11] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-01-08 00:51:13.045815] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=8] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1001, primary_zone=RANDOM) [2025-01-08 00:51:13.045821] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5] add_locality_zone success(locality_zone={tenant_id:1001, region_priority:18446744073709551615}) [2025-01-08 00:51:13.045832] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=4] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1002, primary_zone=RANDOM) [2025-01-08 00:51:13.045838] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5] add_locality_zone success(locality_zone={tenant_id:1002, region_priority:18446744073709551615}) [2025-01-08 00:51:13.045845] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1001, region_priority:18446744073709551615}, {tenant_id:1002, region_priority:18446744073709551615}]}) [2025-01-08 00:51:13.104694] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=34] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:13.110881] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2025-01-08 00:51:13.110925] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=38] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:13.129056] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:13.133170] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:13.133294] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=121][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.136539] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58463][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1C2BB5F8-0-0] [lt=18] init point thread id with(&point=0x5572d50e4100, idx_=1119, point=[thread id=58463, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58463) [2025-01-08 00:51:13.143285] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=106] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:13.180910] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=53] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:13.228962] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58465][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:13.229317] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185122-0-0] [lt=31] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:13.229629] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:13.233447] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.257632] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=34] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:13.261782] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1}, {id:1003}, {id:1001}]) [2025-01-08 00:51:13.266110] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=15] gc stale ls task succ [2025-01-08 00:51:13.267327] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:13.273027] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3AA-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:13.273582] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] ====== check clog disk timer task ====== [2025-01-08 00:51:13.273615] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:13.278417] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:13.286050] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:13.290278] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:202) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=7] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2025-01-08 00:51:13.290310] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=28] finish to resize log external storage handler(current_ts=1736326273290306, tenant_max_cpu=4, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:13.290324] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=12] snapshot_traffic_info once [2025-01-08 00:51:13.291338] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:13.296552] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=49] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:13.307169] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=89] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:13.311722] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=43] gc stale ls task succ [2025-01-08 00:51:13.313987] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:13.314035] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=41] temporary file module use no blocks [2025-01-08 00:51:13.314048] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:13.314062] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:13.314076] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193777227753, flush_io_finished_ret_:0, flush_io_finished_round_:1936, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:13.319004] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.319383] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:13.319698] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.325986] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:13.329229] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:13.333513] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.334746] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=93] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:13.337338] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1736326273337334, bytes: 873215765, bw: 0.241462 MB/s, add_ts: 1004248, add_bytes: 254267 [2025-01-08 00:51:13.342887] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAF-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:13.342932] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CAF-0-0] [lt=41] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:13.343771] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=137] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:13.359269] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:13.359305] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:13.364590] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8708B-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:13.364638] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:13.364650] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:13.365060] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=95] current monitor number(seq_=-1) [2025-01-08 00:51:13.369975] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185424-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:13.371330] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=48] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:13.376118] INFO [STORAGE.TRANS] create_tx_ctx_ (ob_trans_ctx_mgr_v4.cpp:419) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ACFC-0-0] [lt=42] transaction statistics(ls_id={id:1}, total_count=1) [2025-01-08 00:51:13.376373] INFO [STORAGE.TRANS] get_tx_ctx_ (ob_trans_ctx_mgr_v4.cpp:496) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=50] transaction statistics(ls_id={id:1}, total_tx_ctx_count=1) [2025-01-08 00:51:13.387393] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get number(ret=0, service_type_=0, range=1, base_id=1736326273384622000, start_id=1736326273384622000, end_id=1736326273384622001) [2025-01-08 00:51:13.389056] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3082) [2025-01-08 00:51:13.424184] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:13.424498] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=286] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:13.424563] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:13.424579] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:13.426004] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:13.428515] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:13.428705] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D044-0-0] [lt=68][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:13.428775] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4597-0-0] [lt=31] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:13.429154] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4882, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:13.429251] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:13.429273] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=244, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:13.429334] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:13.429350] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6740, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:13.429378] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=34, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:13.429512] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:13.431804] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=334] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4450, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:13.431971] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=165] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:13.431993] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=212, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:13.432071] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:13.433522] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:51:13.433671] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.447319] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22] ====== tenant freeze timer task ====== [2025-01-08 00:51:13.449676] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:13.449842] WDIAG read_multi_row (ob_table_access_helper.h:238) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:13.451630] INFO delete_row (ob_table_access_helper.h:339) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=47] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1001, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326273449891) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326273449891) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:13.451704] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:13.451725] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71333860, tenant_id=1001) [2025-01-08 00:51:13.451745] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:13.459443] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=100] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.459556] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.471208] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=38] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:13.471273] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=57] temporary file module use no blocks [2025-01-08 00:51:13.471287] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:13.471300] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:13.471362] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193777384877, flush_io_finished_ret_:0, flush_io_finished_round_:1932, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:13.471437] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:13.471457] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:51:13.471463] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:13.471471] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:13.471481] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193777384891, flush_io_finished_ret_:0, flush_io_finished_round_:3760, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:13.495736] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=34] clear dtl interm result cost(us)(clear_cost=8627, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:13.512236] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F4E9-0-0] [lt=42] submit log success(service_type_=0, last_id=1736326290087570000, limited_id=1736326290087570000) [2025-01-08 00:51:13.512286] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F4E9-0-0] [lt=49] submit log with lock success(service_type_=0, last_id=1736326290087570000, limited_id=1736326290087570000) [2025-01-08 00:51:13.515824] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=122] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=309, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:13.515920] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=94] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:13.516674] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] update id meta(ret=0, service_type=0, limited_id=1736326290087570000, latest_scn={val:1736326273510997001, v:0}, *this={tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326270511849003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326290087570000, latest_log_ts:{val:1736326273510997001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:13.516743] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=67] handle submit callback(service_type_=0, success=true, used_ts=4556, log_ts={val:1736326273510997001, v:0}, limited_id=1736326290087570000) [2025-01-08 00:51:13.516757] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=13] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:13.516766] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:13.520334] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=46] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.523850] INFO [LIB] stat (utility.h:1210) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4599-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:13.523877] INFO [LIB] stat (utility.h:1210) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4599-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=39489, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:13.523888] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4599-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=18, avg_log_submit_to_flush_time=5171, avg_log_submit_to_slide_time=39471) [2025-01-08 00:51:13.524501] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1339) [2025-01-08 00:51:13.524527] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:13.524537] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=109] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:13.524572] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17984, miss_cnt:0, cache_read_size:2200222, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:13.530487] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:13.533823] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.542290] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.544365] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:13.544507] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=100] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.552281] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3668) [2025-01-08 00:51:13.552380] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=94] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:13.559932] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:13.559992] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:49027, [delay/us]:prepare:8, schedule:97, submit:71, rt:1715, total:1892) [2025-01-08 00:51:13.560009] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:13.560065] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:49029, [delay/us]: prepare:8, schedule:97, submit:71, device:1715, total:1892) [2025-01-08 00:51:13.560130] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=58] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:13.560172] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:18, bw:85798, [delay/us]:prepare:7, schedule:87, submit:663, rt:2088, total:2847) [2025-01-08 00:51:13.560226] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:13.560270] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:18, avg_bw:85794, [delay/us]: prepare:7, schedule:87, submit:663, device:2088, total:2847) [2025-01-08 00:51:13.560282] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:13.560366] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16341, [delay/us]:prepare:8, schedule:21, submit:1, rt:3176, total:3207) [2025-01-08 00:51:13.560393] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4395, iops:41, bw:179748, [delay/us]:prepare:7, schedule:215, submit:692, rt:2652, total:3568) [2025-01-08 00:51:13.560489] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=88] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:13.560545] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16339, [delay/us]: prepare:8, schedule:21, submit:1, device:3176, total:3207) [2025-01-08 00:51:13.560567] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4396, avg_iops:41, avg_bw:179739, [delay/us]: prepare:7, schedule:215, submit:692, device:2652, total:3568) [2025-01-08 00:51:13.560578] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:13.560666] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:418, total:431) [2025-01-08 00:51:13.560740] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:13.560778] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4084, [delay/us]: prepare:0, schedule:0, submit:600000000, device:418, total:431) [2025-01-08 00:51:13.560797] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:13.560814] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:13.560822] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:13.560830] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:13.560838] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=255kB/s, net_out=220kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:13.561868] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15013451155, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:13.561968] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=28] recycle one thread(this=0x2ad8881fc3b0, total=6, remain=5) [2025-01-08 00:51:13.561982] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15013451155, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:13.561993] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7777626258, tenant_id:1001}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:13.562055] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] recycle one thread(this=0x2ad8f90b0e70, total=7, remain=6) [2025-01-08 00:51:13.562066] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7777626258, tenant_id:1001}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:13.581141] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:194) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=47] get wrs ts(ls_id={id:1}, delta=461137, log_service_ts=1736326273118733, min_tx_service_ts=4611686018427387, timestamp={val:1736326273118733003, v:0}) [2025-01-08 00:51:13.581273] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=122] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=83, Other_Error_Cnt=0, Submit_Succ_Cnt=290, last_scn="{val:1736326272117597004, v:0}", last_lsn={lsn:18671144}, last_gts={val:1736326273118733000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-01-08 00:51:13.581347] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1736326273118733003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3, epoch_tstamp:1736326273581080}, version_delta=462613) [2025-01-08 00:51:13.600893] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58466][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:13.601063] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58466][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BB1-0-0] [lt=75] init point thread id with(&point=0x5572d50e41c0, idx_=1122, point=[thread id=58466, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58466) [2025-01-08 00:51:13.603833] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C98-0-0] [lt=150][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:13.604482] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4F-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326273697119, keepalive_ts:-1}) [2025-01-08 00:51:13.604515] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF4F-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326273697119, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:13.605362] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1, group_id_=1, token=6) [2025-01-08 00:51:13.608027] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C98-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:13.609331] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84296049}], scn_array=[cnt:1, {val:1736326273591786000, v:0}], curr_size=586, accum_size=3418, time_cost=16317) [2025-01-08 00:51:13.613494] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58460][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D7A3-0-0] [lt=101] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:13.620023] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:13.629692] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58452][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A459B-0-0] [lt=68] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:13.633961] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.639622] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C62E-0-0] [lt=32][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193778553454, cache_obj->added_lc()=false, cache_obj->get_object_id()=13612, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:13.641154] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43202484}, readable_log_tail:{lsn:43202484}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23902284, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:13.652568] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=33] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326269600101, last_first_register_time_us=1736324454612308, register_time_us=1736326269600101, curr_time_us=1736326273652565) [2025-01-08 00:51:13.652667] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=38] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326273652609, last_submit_register_req_time_us=1736326273652609, register_parent_reason=1) [2025-01-08 00:51:13.652687] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:13.676424] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=30] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:13.676450] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:13.676498] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84982-0-0] [lt=16] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:13.676514] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84982-0-0] [lt=13] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:13.696240] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=52] BlockGCTimerTask success(ret=0, cost_time_us=89, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:13.710754] INFO [LIB] stat (utility.h:1210) [58452][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC383F-0-0] [lt=37] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:13.710961] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58452][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC383F-0-0] [lt=191] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:13.724867] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=107, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:13.724926] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1348, avg_group_log_size=122) [2025-01-08 00:51:13.725410] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=37] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/71:137120 disk:308K/71:137120 t1m2:44K/10:105392 t1m2g0:44K/10:105392 t1002m2:188K/44:170040 t1002m2g0:188K/44:170040 t1001m2:76K/17:70578 t1001m2g0:76K/17:70578 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:13.726918] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3841, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:13.727066] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=146] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1348) [2025-01-08 00:51:13.727090] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=307, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:13.727223] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=51, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:13.729009] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A459D-0-0] [lt=25] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:13.729840] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91B4-0-0] [lt=38] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:13.729863] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91B4-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6230, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:13.730498] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91B5-0-0] [lt=420] init point thread id with(&point=0x5572d50e4180, idx_=1121, point=[thread id=58465, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58465) [2025-01-08 00:51:13.730790] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:13.734078] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.744906] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:13.752430] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=104, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:13.752480] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3540, avg_group_log_size=354) [2025-01-08 00:51:13.753766] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58464][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3941-0-0] [lt=46] init point thread id with(&point=0x5572d50e4140, idx_=1120, point=[thread id=58464, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58464) [2025-01-08 00:51:13.754024] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=17195963, this={dio_aligned_buf:{buf_write_offset:1077, buf_padding_size:3019, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:70613, truncate_used_ts:17408}, log_block_size:67108864, io_fd:{first_id:-1, second_id:152, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=17195008, buf_len=122, total_write_size=64281, total_write_size_after_dio=589824, ob_pwrite_used_ts=2659998, count=128, each_pwrite_cost=20781) [2025-01-08 00:51:13.754066] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3474, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:13.754078] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3540) [2025-01-08 00:51:13.754096] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=120, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:13.754124] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:13.755789] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BB2-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:13.755813] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BB2-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5845, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:13.755823] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BB2-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=126, avg_log_submit_to_flush_time=3735, avg_log_submit_to_slide_time=5718) [2025-01-08 00:51:13.756062] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=92] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3540, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:13.762405] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5232168805, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:13.762785] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=49] recycle one thread(this=0x2ad8885abcf0, total=4, remain=3) [2025-01-08 00:51:13.762804] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5232168805, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:13.765284] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C630-0-0] [lt=52][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193778679116, cache_obj->added_lc()=false, cache_obj->get_object_id()=13613, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:13.777668] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AD02-0-0] [lt=61] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.863697242736816406e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=393, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.343607080527527842e+02) [2025-01-08 00:51:13.778084] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=10] rpc statistics(total_trans_msg_count=39, total_batch_msg_count=0) [2025-01-08 00:51:13.779086] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:263) [50684][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1FD8458E-0-0] [lt=21] ObArchiveSender is running(thread_index=1) [2025-01-08 00:51:13.786256] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=43] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:13.789284] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=40] PNIO [ratelimit] time: 1736326273789281, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007541, add_bytes: 0 [2025-01-08 00:51:13.789704] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=47] ====== check clog disk timer task ====== [2025-01-08 00:51:13.789731] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:13.791583] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=166] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:13.802627] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=36] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:13.805764] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:13.807408] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:13.807560] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:13.807793] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326273384622001, v:0}, new_max_version={val:1736326273384622001, v:0}, last_min_version={val:1736326273186829001, v:0}, last_max_version={val:1736326273186829001, v:0}, total_time_us=17628, delta_us=423168) [2025-01-08 00:51:13.807853] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326273384622001, v:0}, min_version={val:1736326273384622001, v:0}, max_version={val:1736326273384622001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:13.808145] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=645615616, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=428126208, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:13.808289] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=93] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:13.808313] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=22] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2829, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:13.808430] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=66] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:13.808486] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:13.808545] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=57] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:13.814793] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3841-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:13.814838] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3841-0-0] [lt=45] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29928, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:13.814852] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58460][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3841-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=44, avg_log_submit_to_flush_time=2737, avg_log_submit_to_slide_time=29883) [2025-01-08 00:51:13.814865] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58460][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3841-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17976, log_task={header:{begin_lsn:{lsn:2187945}, end_lsn:{lsn:2188067}, log_id:17976, min_scn:{val:1736326273262998004, v:0}, max_scn:{val:1736326273262998004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2187823}, prev_proposal_id:1, committed_end_lsn:{lsn:2187945}, data_checksum:940009244, accum_checksum:3298465875, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326273710738, freeze_ts:1736326273710738, submit_ts:1736326273710950, flushed_ts:1736326273716263, gen_to_freeze cost time:0, gen_to_submit cost time:212, submit_to_flush cost time:5313}, fs_cb_begin_ts=1736326273814780, log_life_time=104042) [2025-01-08 00:51:13.815024] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=104] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:13.815054] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17886, miss_cnt:0, cache_read_size:2188067, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:13.819514] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:13.822106] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58468][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:13.822110] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-01-08 00:51:13.830621] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=249] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=70, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:13.834753] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.835612] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=53] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:13.836271] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9637) [2025-01-08 00:51:13.869604] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=9] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:13.896104] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F8-0-0] [lt=93] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:13.896161] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F8-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326273262998004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326273262998004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:13.897289] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.897442] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.897488] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.897512] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:13.912043] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2657, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:13.912076] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:13.912096] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=136, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:13.912154] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=72, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:13.916610] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F8-0-0] [lt=52] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=9637, avg_group_log_size=566) [2025-01-08 00:51:13.916642] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5F8-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9637) [2025-01-08 00:51:13.918310] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=2218, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:13.918349] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10387) [2025-01-08 00:51:13.918369] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=458, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:13.918402] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=26, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:13.918586] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58462][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC2-0-0] [lt=35] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326274013663, keepalive_ts:-1}) [2025-01-08 00:51:13.918616] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58462][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC2-0-0] [lt=28] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326274013663, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:13.920135] INFO [LIB] stat (utility.h:1210) [58462][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2CAC-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=14, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:13.920158] INFO [LIB] stat (utility.h:1210) [58462][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2CAC-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=5699, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:13.920168] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58462][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2CAC-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=92, avg_log_submit_to_flush_time=2711, avg_log_submit_to_slide_time=5607) [2025-01-08 00:51:13.920348] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10387, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:13.920407] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=26] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:13.929173] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8582-0-0] [lt=51] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=150, avg_log_submit_to_flush_time=3258, avg_log_submit_to_slide_time=5120) [2025-01-08 00:51:13.929614] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1470, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:13.932081] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=59] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1226784, replace_num=15728) [2025-01-08 00:51:13.932170] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=86] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:13.936365] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:13.945436] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=94] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:13.953284] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:19440, max_lsn:18673037, max_scn:{val:1736326273619930003, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:18673037}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:18672915}, last_submit_end_lsn:{lsn:18673037}, last_submit_log_id:19440, last_submit_log_pid:1, max_flushed_lsn:{lsn:18672915}, max_flushed_end_lsn:{lsn:18673037}, max_flushed_log_pid:1, committed_end_lsn:{lsn:18673037}, last_slide_log_id:19440, last_slide_scn:{val:1736326273619930003, v:0}, last_slide_lsn:{lsn:18672915}, last_slide_end_lsn:{lsn:18673037}, last_slide_log_pid:1, last_slide_log_accum_checksum:2193338477, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8fc454530}, [StateMgr]={this:0x2ad8fc458830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:19441, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324399554770, palf_role_change_cb:0x2ad8fc45f2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8fc455c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324399758968, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/meta", dir_fd:175, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8fc45aa38, flashback_version:0}, log_storage:{log_tail:{lsn:18673037}, readable_log_tail:{lsn:18673037}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48431731, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324399554279, this:0x2ad8fc458f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324399555093, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8fc4589b0}) [2025-01-08 00:51:13.953613] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=322] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326273927595, last_flushed_end_lsn:{lsn:18673037}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326273934320, last_flushed_end_lsn:{lsn:18673037}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326273929069, last_flushed_end_lsn:{lsn:18673037}}]) [2025-01-08 00:51:13.953641] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=19441, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326273927595, last_flushed_end_lsn:{lsn:18673037}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326273934320, last_flushed_end_lsn:{lsn:18673037}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326273929069, last_flushed_end_lsn:{lsn:18673037}}]) [2025-01-08 00:51:13.953683] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1002, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18081, max_lsn:2200832, max_scn:{val:1736326273510997004, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2200832}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2200710}, last_submit_end_lsn:{lsn:2200832}, last_submit_log_id:18081, last_submit_log_pid:1, max_flushed_lsn:{lsn:2200710}, max_flushed_end_lsn:{lsn:2200832}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2200832}, last_slide_log_id:18081, last_slide_scn:{val:1736326273510997004, v:0}, last_slide_lsn:{lsn:2200710}, last_slide_end_lsn:{lsn:2200832}, last_slide_log_pid:1, last_slide_log_accum_checksum:978508813, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad92435e530}, [StateMgr]={this:0x2ad924362830, self:"192.168.159.137:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.138:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18082, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad9243692f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326272645954, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326272645954, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad92435fc70}, [ModeMgr]={palf_id:1002, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436803442001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/meta", dir_fd:184, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924364a38, flashback_version:0}, log_storage:{log_tail:{lsn:2200832}, readable_log_tail:{lsn:2200832}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64903936, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:-1, this:0x2ad924362f70}, [Reconfirm]={palf_id:1002, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9243629b0}) [2025-01-08 00:51:13.953889] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=201][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:13.953905] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:13.953980] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=71][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:13.953992] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] current log_task status(palf_id=1002, self="192.168.159.137:2882", log_id=18082, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:51:13.972384] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7808098926, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:13.972516] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=57] recycle one thread(this=0x2ad8c0eb0e70, total=5, remain=4) [2025-01-08 00:51:13.972531] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7808098926, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:13.973487] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=42] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:13.976977] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=60] get number(ret=0, service_type_=0, range=1, base_id=1736326273974298000, start_id=1736326273974298000, end_id=1736326273974298001) [2025-01-08 00:51:13.977401] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=28] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:14.017015] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8644, offset=2453170}, end_cursor:ObLogCursor{file_id=1, log_id=8645, offset=2453475}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8947fc050, len:4430, log_data_len:305, seq:8644, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:14.017591] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4F-0-0] [lt=117] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:14.017657] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4F-0-0] [lt=44] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326273619930003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326273619930003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:14.036234] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:14.036570] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=79] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8645, offset=2453475}, end_cursor:ObLogCursor{file_id=1, log_id=8646, offset=2453772}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a04050, len:4125, log_data_len:297, seq:8645, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:14.036618] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:14.037149] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58464][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=70] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:51:14.037698] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4F-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:14.037778] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E4F-0-0] [lt=48] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326273619930003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326273619930003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:14.042315] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B9D-0-0] [lt=36] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:14.050376] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8646, offset=2453772}, end_cursor:ObLogCursor{file_id=1, log_id=8647, offset=2454069}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a08050, len:3828, log_data_len:297, seq:8646, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:14.051695] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=56] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8647, offset=2454069}, end_cursor:ObLogCursor{file_id=1, log_id=8648, offset=2454366}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a10050, len:3531, log_data_len:297, seq:8647, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:14.067306] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A2-0-0] [lt=38] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:14.067424] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A2-0-0] [lt=86] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326274012652000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326274012652000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:14.104908] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=84] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:14.108852] INFO [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1165) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] refresh queuing mode(ret=0, tenant_id=1001, stream_cnt=0, update_schema_cnt=0, cost_time=131) [2025-01-08 00:51:14.108888] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1215) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=33] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1001, interval_step=1, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1001, bucket_cnt:0, tenant_mem_user_limit:0, tenant_mem_user_hold:0, cached_node_array:0x2ad8f95d0ee0}) [2025-01-08 00:51:14.121340] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=43] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=468, get_gts_with_stc_cnt=3, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=3, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:14.132807] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=148] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:14.136792] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=405][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:14.145953] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=67] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:14.147985] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=38] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=723511, clean_num=31457) [2025-01-08 00:51:14.233556] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:14.233695] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58452][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=19] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:14.236910] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:14.257795] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:14.261974] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1003}, {id:1}]) [2025-01-08 00:51:14.267316] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:51:14.267489] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:14.272680] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3AD-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:14.275835] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=49] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326273619930004, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=655893, in_cluster_service=true, cluster_version={val:1736326273510997004, v:0}, min_cluster_version={val:1736326273510997004, v:0}, max_cluster_version={val:1736326273510997004, v:0}, get_cluster_version_err=0, cluster_version_delta=764826, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18291, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=502193350, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:14.286470] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=31] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:14.292126] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:14.300923] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD8-0-0] [lt=85] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:14.301009] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD8-0-0] [lt=32] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=44) [2025-01-08 00:51:14.303411] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=186] LogLoopThread round_cost_time(us)(round_cost_time=13) [2025-01-08 00:51:14.307578] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:14.307722] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58424][T1001_TimerWK6][T1001][Y0-0000000000000000-0-0] [lt=106] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:14.311148] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=91] blacklist_loop exec finished(cost_time=34, is_enabled=true, send_cnt=0) [2025-01-08 00:51:14.311845] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=46] gc stale ls task succ [2025-01-08 00:51:14.314857] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:14.314903] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=40] temporary file module use no blocks [2025-01-08 00:51:14.314916] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:14.314929] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:14.314943] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193778227948, flush_io_finished_ret_:0, flush_io_finished_round_:1937, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:14.318993] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:14.319730] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=76] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:14.319990] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:14.328948] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:14.329871] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58469][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:14.331047] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:14.333324] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:14.335849] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58423][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:14.338716] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:14.339519] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1736326274339517, bytes: 873452436, bw: 0.225215 MB/s, add_ts: 1002183, add_bytes: 236671 [2025-01-08 00:51:14.345074] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB0-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:14.345150] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB0-0-0] [lt=72] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:14.348453] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:14.353904] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] LogLoopThread round_cost_time(us)(round_cost_time=19) [2025-01-08 00:51:14.360507] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:14.360547] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:14.361516] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=31] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:14.361915] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=393] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:14.361930] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:14.361942] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:14.365561] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:14.366641] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=1074] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:14.371054] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=29] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:14.372674] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=38] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:14.372880] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=78] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326274317657000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:14.373012] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=89] current monitor number(seq_=-1) [2025-01-08 00:51:14.373385] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:14.375082] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185425-0-0] [lt=102][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:14.375426] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=35] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:14.375548] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=37] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326265706032000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:14.375576] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326265706032000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326265706032000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:14.378547] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=21] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:14.383847] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=46] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:14.386480] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87091-0-0] [lt=27][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:14.386558] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:14.386573] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:14.395906] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6851-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4105) [2025-01-08 00:51:14.407262] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1736326274189523000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84300032}, palf_committed_end_scn_:{val:1736326274393594000, v:0}, last_check_scn_:{val:1736326274393594000, v:0}, max_applied_cb_scn_:{val:1736326274189523000, v:0}}) [2025-01-08 00:51:14.407308] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84300032}, readable_log_tail:{lsn:84300032}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49909504, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:14.415037] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=115] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:14.421361] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=82] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:14.421660] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=61] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326265595928000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:14.425112] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58469][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D7B3-0-0] [lt=65] init point thread id with(&point=0x5572d50e4280, idx_=1125, point=[thread id=58469, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58469) [2025-01-08 00:51:14.426850] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:14.426891] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:14.426989] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:14.427028] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:14.430643] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2010, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:14.430691] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:14.430716] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=174, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:14.430762] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:14.431065] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1348) [2025-01-08 00:51:14.431706] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45AB-0-0] [lt=92] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:14.436279] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326274427888000, start_id=1736326274427888000, end_id=1736326274427888001) [2025-01-08 00:51:14.436639] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=125] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:14.436688] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:51:14.437116] INFO [LIB] stat (utility.h:1210) [58389][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8590-0-0] [lt=31] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:14.437133] INFO [LIB] stat (utility.h:1210) [58389][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8590-0-0] [lt=15] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5433, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:14.437991] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:14.438925] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:14.439858] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58470][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:14.439922] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:14.441224] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3173, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:14.441255] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:14.441279] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1980, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:14.441364] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=60, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:14.443874] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50625][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D047-0-0] [lt=5][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:14.448323] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=66] get_max_applied_scn(scn={val:1736326274322331001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43212030}, palf_committed_end_scn_:{val:1736326274427888000, v:0}, last_check_scn_:{val:1736326274427888000, v:0}, max_applied_cb_scn_:{val:1736326274322331001, v:0}}) [2025-01-08 00:51:14.450975] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=81] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:14.451029] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=51] try gc retain ctx [2025-01-08 00:51:14.451040] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-01-08 00:51:14.451087] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326274427888000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326263763810001, v:0}, keep_alive_scn_:{val:1736326263763810001, v:0}, update_ts_:1736326274451061}) [2025-01-08 00:51:14.452087] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=26] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289284252000, latest_log_ts:{val:1736326272625185001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:14.453258] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:14.453284] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] try gc retain ctx [2025-01-08 00:51:14.453291] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] try to retry start_working [2025-01-08 00:51:14.453368] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=46] finish update min start scn(max_decided_scn={val:1736326274393594000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326263731517001, v:0}, keep_alive_scn_:{val:1736326263731517001, v:0}, update_ts_:1736326274452223}) [2025-01-08 00:51:14.454343] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289254996000, latest_log_ts:{val:1736326272617414001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:14.462046] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:14.462277] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=60] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:14.493308] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=74] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:14.519231] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:14.527083] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:14.527170] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:14.533817] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:14.537692] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D7B6-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=33, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:14.537740] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D7B6-0-0] [lt=46] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=23157, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:14.537755] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58460][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D7B6-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=16, avg_log_submit_to_flush_time=4977, avg_log_submit_to_slide_time=23141) [2025-01-08 00:51:14.537868] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:14.537904] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17995, miss_cnt:0, cache_read_size:2201564, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:14.537972] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=115, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:14.537985] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=13] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:14.539149] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:14.541922] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:14.544568] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:14.548938] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:14.553483] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=136] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4218) [2025-01-08 00:51:14.553511] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:14.562708] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:14.562761] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:53100, [delay/us]:prepare:8, schedule:137, submit:72, rt:2662, total:2881) [2025-01-08 00:51:14.562776] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:14.562832] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:53101, [delay/us]: prepare:8, schedule:137, submit:72, device:2662, total:2881) [2025-01-08 00:51:14.562845] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:14.562884] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4527, iops:19, bw:85783, [delay/us]:prepare:11, schedule:89, submit:120, rt:2065, total:2287) [2025-01-08 00:51:14.562894] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:14.562930] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4527, avg_iops:19, avg_bw:85787, [delay/us]: prepare:11, schedule:89, submit:120, device:2065, total:2287) [2025-01-08 00:51:14.562940] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:14.562962] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:6144, iops:4, bw:24510, [delay/us]:prepare:6, schedule:666, submit:1, rt:1876, total:2550) [2025-01-08 00:51:14.562984] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4198, iops:40, bw:167500, [delay/us]:prepare:10, schedule:125, submit:112, rt:1877, total:2126) [2025-01-08 00:51:14.562993] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:14.563021] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:6144, avg_iops:4, avg_bw:24515, [delay/us]: prepare:6, schedule:666, submit:1, device:1876, total:2550) [2025-01-08 00:51:14.563036] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4198, avg_iops:40, avg_bw:167522, [delay/us]: prepare:10, schedule:125, submit:112, device:1877, total:2126) [2025-01-08 00:51:14.563046] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:14.563064] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4086, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1111, total:1121) [2025-01-08 00:51:14.563087] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:14.563168] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4086, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1111, total:1121) [2025-01-08 00:51:14.563256] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:14.563276] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:14.563285] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:14.563293] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:14.563300] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=275kB/s, net_out=230kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:14.576501] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15018502205, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:14.576539] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:14.576552] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8881fc3b0, thread=0x2ad8aedf79c0) [2025-01-08 00:51:14.576561] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15018502205, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:14.576617] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=55] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7783732768, tenant_id:1001}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:14.576630] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:14.576638] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=7] try_wait return eagain(this=0x2ad8f90b0e70, thread=0x2ad90d0bf610) [2025-01-08 00:51:14.576644] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7783732768, tenant_id:1001}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:14.602383] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46382][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BEC9-0-0] [lt=38] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:14.616875] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C99-0-0] [lt=96][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:14.619685] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C99-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:14.621727] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=50] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:14.629086] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58469][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45AF-0-0] [lt=280] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:14.629439] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [50591][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=20] XA scheduler heartbeat task statistics(avg_time=0) [2025-01-08 00:51:14.639017] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8597-0-0] [lt=127] init point thread id with(&point=0x5572d50e42c0, idx_=1126, point=[thread id=58470, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58470) [2025-01-08 00:51:14.639297] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:14.650558] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B9F-0-0] [lt=25] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:14.662865] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C640-0-0] [lt=36][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193779576695, cache_obj->added_lc()=false, cache_obj->get_object_id()=13614, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:14.673772] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=18] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:14.683322] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=123] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:14.692450] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE2D-0-0] [lt=16] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:14.692512] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE2D-0-0] [lt=53] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:14.692547] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE2D-0-0] [lt=32] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:14.714491] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3853-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:14.714815] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58460][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3853-0-0] [lt=306] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:14.714838] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58460][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3853-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:14.727080] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=147] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=108, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:14.727119] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:14.727453] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:332K/76:133639 disk:332K/76:133639 t1m2:52K/12:124908 t1m2g0:52K/12:124908 t1002m2:192K/45:158427 t1002m2g0:192K/45:158427 t1001m2:88K/19:80446 t1001m2g0:88K/19:80446 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:14.728619] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1938, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:14.728648] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:14.728740] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=353, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:14.728739] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:14.729553] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=33] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:14.731498] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC859A-0-0] [lt=41] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:14.731520] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC859A-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5749, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:14.731590] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45B1-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:14.731897] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=94] get number(ret=0, service_type_=0, range=1, base_id=1736326274729532000, start_id=1736326274729532000, end_id=1736326274729532001) [2025-01-08 00:51:14.733314] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=159] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326274427888001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=305414, in_cluster_service=true, cluster_version={val:1736326274427888000, v:0}, min_cluster_version={val:1736326274427888000, v:0}, max_cluster_version={val:1736326274427888000, v:0}, get_cluster_version_err=0, cluster_version_delta=305414, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18730, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=620370085, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:14.734172] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:14.739410] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:14.747743] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=24] ====== tenant freeze timer task ====== [2025-01-08 00:51:14.749788] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:14.749855] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:14.749893] WDIAG read_multi_row (ob_table_access_helper.h:238) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:14.751771] INFO delete_row (ob_table_access_helper.h:339) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326274749911) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326274749911) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:14.753692] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=91, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:14.753735] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=4218, avg_group_log_size=351) [2025-01-08 00:51:14.755793] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1835, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:14.755823] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4218) [2025-01-08 00:51:14.755843] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=138, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:14.756402] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:14.761952] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D3C-0-0] [lt=32] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:14.765042] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58471][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:14.765113] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-01-08 00:51:14.765138] INFO [LIB] stat (utility.h:1210) [58436][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6856-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:14.765160] INFO [LIB] stat (utility.h:1210) [58436][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6856-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=6024, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:14.765171] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58436][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6856-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=119, avg_log_submit_to_flush_time=2061, avg_log_submit_to_slide_time=5905) [2025-01-08 00:51:14.765330] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=54] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4218, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:14.767280] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE856-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326274766923000, start_id=1736326274766923000, end_id=1736326274766923001) [2025-01-08 00:51:14.767637] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE855-0-0] [lt=105] get number(ret=0, service_type_=0, range=1, base_id=1736326274766923000, start_id=1736326274766923000, end_id=1736326274766923001) [2025-01-08 00:51:14.775684] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C642-0-0] [lt=46][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193779689516, cache_obj->added_lc()=false, cache_obj->get_object_id()=13615, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:14.776826] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5235041018, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:14.776860] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:14.776872] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8df7f30f0) [2025-01-08 00:51:14.776881] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5235041018, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:14.779176] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AD11-0-0] [lt=58] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=156) [2025-01-08 00:51:14.779266] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AD11-0-0] [lt=28] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.863697242736816406e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=393, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.343607080527527842e+02) [2025-01-08 00:51:14.782980] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=14] [APPLY STAT CB APPEND COST TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=139, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:14.783007] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=27] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=8106, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:14.783017] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:14.783025] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=-1933, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:14.783032] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=6312, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:14.786906] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=65] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:14.788361] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:219) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:14.788923] INFO [COMMON] handle (memory_dump.cpp:538) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=560] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:14.790526] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] update_check_range(min_check_version=373, max_check_version=374, global_version=375) [2025-01-08 00:51:14.799239] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1736326274799235, bytes: 0, bw: 0.000000 MB/s, add_ts: 1009954, add_bytes: 0 [2025-01-08 00:51:14.800942] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=36] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:14.802323] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326274799332000, start_id=1736326274799332001, end_id=1736326274799332002) [2025-01-08 00:51:14.802551] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:14.807596] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=27] ====== checkpoint timer task ====== [2025-01-08 00:51:14.808534] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:14.808619] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=84] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:14.808726] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289284252000, latest_log_ts:{val:1736326272625185001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:14.808808] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=78] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:14.808839] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:14.808852] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326272625185001, v:0}) [2025-01-08 00:51:14.808860] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:14.808868] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:14.808879] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:14.808896] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:14.808909] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:14.808930] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:14.808951] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:14.808960] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:14.809511] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=49] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:14.811759] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=19] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:14.812462] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=18] succ to submit log task to replay service(to_submit_lsn={lsn:2189212}, to_submit_scn={val:1736326274264437004, v:0}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326274812304, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2189278}, next_to_submit_scn_:{val:1736326274264437005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2189156}, end_lsn:{lsn:2189278}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2189278}, readable_log_tail:{lsn:2189278}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64915490, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326274264437004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326274264437004, v:0}, data_checksum:2124447694, flag:64015}}, init_mode_version:0, accumulate_checksum:337734816, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326274264437004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17896, miss_cnt:0, cache_read_size:2189278, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:14.815682] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=191] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326274598419000, v:0}, new_max_version={val:1736326274598419000, v:0}, last_min_version={val:1736326274393594000, v:0}, last_max_version={val:1736326274393594000, v:0}, total_time_us=17038, delta_us=217261) [2025-01-08 00:51:14.815716] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326274598419000, v:0}, min_version={val:1736326274598419000, v:0}, max_version={val:1736326274598419000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:14.816989] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91D5-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=28, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:14.817015] INFO [LIB] stat (utility.h:1210) [58460][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91D5-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21059, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:14.817019] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:14.817044] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17897, miss_cnt:0, cache_read_size:2189400, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:14.817024] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58460][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91D5-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=43, avg_log_submit_to_flush_time=2214, avg_log_submit_to_slide_time=21015) [2025-01-08 00:51:14.817120] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326274816805000, start_id=1736326274816805000, end_id=1736326274816805001) [2025-01-08 00:51:14.820089] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=6] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:14.830448] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=21] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:14.832840] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45B3-0-0] [lt=42][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18089, log_task={header:{begin_lsn:{lsn:2201686}, end_lsn:{lsn:2201808}, log_id:18089, min_scn:{val:1736326274512770002, v:0}, max_scn:{val:1736326274512770002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2201564}, prev_proposal_id:1, committed_end_lsn:{lsn:2201686}, data_checksum:1715074492, accum_checksum:922244866, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326274731584, freeze_ts:1736326274731584, submit_ts:1736326274731623, flushed_ts:1736326274734540, gen_to_freeze cost time:0, gen_to_submit cost time:39, submit_to_flush cost time:2917}, fs_cb_begin_ts=1736326274832827, log_life_time=101243) [2025-01-08 00:51:14.833006] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=107] get number(ret=0, service_type_=0, range=1, base_id=1736326274832012000, start_id=1736326274832012000, end_id=1736326274832012001) [2025-01-08 00:51:14.833138] INFO [LIB] stat (utility.h:1210) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=65, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:14.836172] INFO handle (memory_dump.cpp:665) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=44] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 151 261 0 1 2 68 1 0 1 4 1 1 0 1 5 43 5 0 1 7 1 2 0 1 8 49 1 0 1 12 7 3 0 1 13 1 1 0 1 14 1 2 0 1 15 1 1 0 1 18 4 3 0 1 20 1 2 0 500 0 94 237 0 500 7 7 4 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 2 1 0 500 17 9 7 0 500 22 79 62 0 500 23 23 10 0 508 0 2 10 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 98 230 0 1001 2 49 1 0 1001 4 1 1 0 1001 5 22 5 0 1001 8 30 1 0 1001 12 3 2 0 1001 13 1 1 0 1001 14 1 1 0 1001 18 4 3 0 1001 20 2 2 0 1002 0 162 245 0 1002 2 59 1 0 1002 4 2 1 0 1002 5 49 6 0 1002 7 11 2 0 1002 8 47 1 0 1002 12 2 3 0 1002 13 1 1 0 1002 14 1 1 0 1002 16 1 1 0 1002 18 9 3 0 1002 20 1 2 0 cost_time: 45683 [2025-01-08 00:51:14.836283] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=85] [MEMORY] tenant: 1, limit: 2,147,483,648 hold: 1,042,788,352 cache_hold: 213,909,504 cache_used: 213,909,504 cache_item_count: 102 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 445,685,760 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 142,606,336 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 95,862,784 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 102,760,448 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 14,680,064 limit= 429,496,720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:14.836686] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 445,685,760 used= 342,145,040 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,211,520 used= 31,196,160 count= 15 avg_used= 2,079,744 block_cnt= 15 chunk_cnt= 15 mod=CACHE_MAP_NODE [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 24,518,656 used= 24,371,776 count= 23 avg_used= 1,059,642 block_cnt= 23 chunk_cnt= 17 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,196,912 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 37 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,268,096 used= 11,228,160 count= 4 avg_used= 2,807,040 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,618,560 used= 7,541,760 count= 10 avg_used= 754,176 block_cnt= 10 chunk_cnt= 8 mod=ResourceGroup [MEMORY] hold= 7,299,072 used= 6,275,072 count= 125 avg_used= 50,200 block_cnt= 125 chunk_cnt= 23 mod=[T]char [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMemCU [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,465,216 used= 3,036,576 count= 141 avg_used= 21,536 block_cnt= 141 chunk_cnt= 30 mod=DiagnosticInfo [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,780,352 used= 2,663,280 count= 586 avg_used= 4,544 block_cnt= 570 chunk_cnt= 29 mod=WorkerMap [MEMORY] hold= 2,768,896 used= 2,720,000 count= 13 avg_used= 209,230 block_cnt= 13 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,640 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 272 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,394,368 used= 1,372,072 count= 75 avg_used= 18,294 block_cnt= 75 chunk_cnt= 11 mod=TabletMap [MEMORY] hold= 1,374,720 used= 1,308,288 count= 36 avg_used= 36,341 block_cnt= 36 chunk_cnt= 13 mod=CommSysVarFac [MEMORY] hold= 1,327,104 used= 1,322,784 count= 9 avg_used= 146,976 block_cnt= 9 chunk_cnt= 7 mod=SQLSessionInfo [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 960,192 used= 936,448 count= 102 avg_used= 9,180 block_cnt= 102 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 950,272 used= 466,784 count= 58 avg_used= 8,048 block_cnt= 58 chunk_cnt= 18 mod=[T]ObPerfEventR [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 516,224 used= 495,296 count= 109 avg_used= 4,544 block_cnt= 109 chunk_cnt= 24 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 262,144 used= 150,272 count= 16 avg_used= 9,392 block_cnt= 16 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 262,144 used= 261,120 count= 4 avg_used= 65,280 block_cnt= 4 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 203,840 used= 194,432 count= 49 avg_used= 3,968 block_cnt= 49 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 12 mod=SqlSession [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 13 mod=LinearHashMapNo [MEMORY] hold= 155,024 used= 2,177 count= 1,920 avg_used= 1 block_cnt= 19 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 154,432 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 13 mod=QueryAllocator [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,456 used= 131,072 count= 2 avg_used= 65,536 block_cnt= 2 chunk_cnt= 2 mod=MinorMergeMgr [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 105,664 used= 103,168 count= 13 avg_used= 7,936 block_cnt= 13 chunk_cnt= 8 mod=SqlSessiVarMap [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 7 mod=PlanVaIdx [MEMORY] hold= 95,792 used= 24,848 count= 352 avg_used= 70 block_cnt= 84 chunk_cnt= 29 mod=Coro [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 5 mod=HashNodeLCSta [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 54,720 used= 51,840 count= 15 avg_used= 3,456 block_cnt= 12 chunk_cnt= 11 mod=CACHE_INST [MEMORY] hold= 53,600 used= 41,664 count= 62 avg_used= 672 block_cnt= 61 chunk_cnt= 21 mod=[T]ObWarningBuf [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 42,960 used= 30,016 count= 67 avg_used= 448 block_cnt= 36 chunk_cnt= 12 mod=OMT_Worker [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 38,960 used= 4,128 count= 172 avg_used= 24 block_cnt= 70 chunk_cnt= 27 mod=[T]MemoryContex [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PartCtx [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,640 used= 15,872 count= 4 avg_used= 3,968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnMemNode [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,640 used= 6,320 count= 12 avg_used= 526 block_cnt= 12 chunk_cnt= 8 mod=SqlSessiQuerSql [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=SqlSessionSbloc [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 7,504 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 7 mod=PackStateMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 8 mod=ContextsMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 6 mod=SequenceMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 5 mod=SequenceIdMap [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,848 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 5 mod=DICollector [MEMORY] hold= 4,864 used= 4,288 count= 3 avg_used= 1,429 block_cnt= 2 chunk_cnt= 2 mod=NegParamIdex [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 2 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 352 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSIDDLVar [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 342,145,040 used= 334,502,002 count= 31,631 avg_used= 10,575 mod=SUMMARY [2025-01-08 00:51:14.836842] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=135] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 142,606,336 used= 141,492,224 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 block_cnt= 68 chunk_cnt= 68 mod=Memstore [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:14.836866] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:14.836887] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 95,862,784 used= 90,371,200 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83,884,032 used= 75,811,934 count= 2,908 avg_used= 26,070 block_cnt= 2,908 chunk_cnt= 43 mod=SqlPhyPlan [MEMORY] hold= 4,709,056 used= 4,468,656 count= 313 avg_used= 14,276 block_cnt= 313 chunk_cnt= 41 mod=SqlPlanCache [MEMORY] hold= 1,353,472 used= 1,234,048 count= 622 avg_used= 1,984 block_cnt= 208 chunk_cnt= 39 mod=CreateContext [MEMORY] hold= 422,656 used= 412,672 count= 52 avg_used= 7,936 block_cnt= 52 chunk_cnt= 12 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 90,371,200 used= 81,928,910 count= 3,897 avg_used= 21,023 mod=SUMMARY [2025-01-08 00:51:14.836907] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,368 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,272 used= 58,751 count= 259 avg_used= 226 block_cnt= 21 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,368 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-01-08 00:51:14.836922] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 102,760,448 used= 100,122,624 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 100,122,624 used= 99,936,384 count= 194 avg_used= 515,136 block_cnt= 194 chunk_cnt= 49 mod=CoStack [MEMORY] hold= 100,122,624 used= 99,936,384 count= 194 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:14.836944] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 213,909,504 used= 213,909,504 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:14.836961] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 14,680,064 used= 12,713,920 limit= 429,496,720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,036,352 used= 7,149,419 count= 109 avg_used= 65,591 block_cnt= 109 chunk_cnt= 5 mod=L_TabletPool [MEMORY] hold= 4,587,520 used= 4,569,600 count= 70 avg_used= 65,280 block_cnt= 70 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 12,713,920 used= 11,803,787 count= 181 avg_used= 65,214 mod=SUMMARY [2025-01-08 00:51:14.836976] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,584,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 block_cnt= 195 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:14.836991] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 16,256 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ExtInfoLog [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:14.837098] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 33,216 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 mod=SUMMARY [2025-01-08 00:51:14.837125] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 14,009,408 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 9,473,984 used= 9,445,184 count= 3 avg_used= 3,148,394 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 4,527,296 used= 4,420,352 count= 557 avg_used= 7,936 block_cnt= 557 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 14,009,408 used= 13,873,472 count= 561 avg_used= 24,729 mod=SUMMARY [2025-01-08 00:51:14.837143] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 73,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 73,152 used= 71,424 count= 9 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:14.837179] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 629,149,696 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 232,894,464 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 24,260,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 170,676,224 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98,557,952 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:14.837574] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=89] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 232,894,464 used= 186,008,976 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,397,376 used= 7,340,256 count= 7 avg_used= 1,048,608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7,259,504 used= 6,324,350 count= 1,032 avg_used= 6,128 block_cnt= 540 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,783,552 used= 5,752,960 count= 6 avg_used= 958,826 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 16 mod=Iteratorget_ls_id()={id:1}) [2025-01-08 00:51:14.900905] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:14.913456] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1749, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:14.913506] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:14.913575] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=178, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:14.913663] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=92, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:14.929538] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F9-0-0] [lt=105] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:14.929606] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5F9-0-0] [lt=45] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326274766923001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326274766923001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:14.934419] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=90] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:14.940151] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58469][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91D8-0-0] [lt=44] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=15, avg_log_gen_to_submit_time=114, avg_log_submit_to_flush_time=2301, avg_log_submit_to_slide_time=5915) [2025-01-08 00:51:14.940982] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1643, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:14.943741] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:14.951887] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=45] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:14.954586] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=34] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12808835072, fast_delete_log_mode=false) [2025-01-08 00:51:14.954684] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=173] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=20, avg_log_batch_cnt=1, total_group_log_size=10878, avg_group_log_size=543) [2025-01-08 00:51:14.954779] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=90] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11506) [2025-01-08 00:51:14.957124] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=58] [PALF STAT WRITE LOG COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=2715, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:14.957143] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10878) [2025-01-08 00:51:14.957159] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=250, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:14.957339] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=30, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:14.957544] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58459][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BC1-0-0] [lt=45] init point thread id with(&point=0x5572d50e4000, idx_=1115, point=[thread id=58459, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58459) [2025-01-08 00:51:14.958312] INFO [LIB] stat (utility.h:1210) [58404][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3958-0-0] [lt=139] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=26, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:14.958336] INFO [LIB] stat (utility.h:1210) [58404][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3958-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=5819, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:14.958345] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58404][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3958-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=20, avg_log_gen_to_freeze_time=421, avg_log_gen_to_submit_time=503, avg_log_submit_to_flush_time=3069, avg_log_submit_to_slide_time=5316) [2025-01-08 00:51:14.958429] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10878, hit_cnt=20, read_cnt=20, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:14.960836] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=39] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:14.962454] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:14.999148] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1736326274799332002, v:0}, base_version={val:1736326274598419000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326274799332002, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326274999129, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326274799332002, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326274994499, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326274799332002, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326275054258, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:15.003729] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=20] [APPLY STAT CB APPEND COST TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=156, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:15.003770] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=40] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=8167, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:15.003781] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:15.003789] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=-834, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:15.003797] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=7489, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:15.037157] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=15] [APPLY STAT CB APPEND COST TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=163, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:15.037281] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=123] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=9381, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:15.037412] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:15.037429] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=16] [APPLY STAT CB EXECUTE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=-1858, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:15.037436] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=7686, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:15.037426] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:15.040432] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58462][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=171] worker thread exit(tenant_->id()=1001, workers_.get_size()=3) [2025-01-08 00:51:15.043938] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.054944] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8648, offset=2454366}, end_cursor:ObLogCursor{file_id=1, log_id=8649, offset=2454671}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a14050, len:3234, log_data_len:305, seq:8648, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:15.055021] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=35] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:32632, max_lsn:43217338, max_scn:{val:1736326275031567000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:43217338}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:1274298}}, last_submit_lsn:{lsn:43216749}, last_submit_end_lsn:{lsn:43217338}, last_submit_log_id:32632, last_submit_log_pid:1, max_flushed_lsn:{lsn:43216749}, max_flushed_end_lsn:{lsn:43217338}, max_flushed_log_pid:1, committed_end_lsn:{lsn:43217338}, last_slide_log_id:32632, last_slide_scn:{val:1736326275031567000, v:0}, last_slide_lsn:{lsn:43216749}, last_slide_end_lsn:{lsn:43217338}, last_slide_log_pid:1, last_slide_log_accum_checksum:56392895, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad9083ee530}, [StateMgr]={this:0x2ad9083f2830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:32633, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324348477669, palf_role_change_cb:0x2ad9083f92f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad9083efc70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324348678844, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/meta", dir_fd:172, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad9083f4a38, flashback_version:0}, log_storage:{log_tail:{lsn:43217338}, readable_log_tail:{lsn:43217338}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23887430, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324348477523, this:0x2ad9083f2f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.138:2882";"192.168.159.139:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324348478003, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9083f29b0}) [2025-01-08 00:51:15.055411] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E50-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:15.055397] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=369] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326275038768, last_flushed_end_lsn:{lsn:43217338}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326275042883, last_flushed_end_lsn:{lsn:43217338}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326275042724, last_flushed_end_lsn:{lsn:43217338}}]) [2025-01-08 00:51:15.055448] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E50-0-0] [lt=17] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326274729532001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326274729532001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:15.055470] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=67] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=32633, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326275038768, last_flushed_end_lsn:{lsn:43217338}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326275042883, last_flushed_end_lsn:{lsn:43217338}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326275042724, last_flushed_end_lsn:{lsn:43217338}}]) [2025-01-08 00:51:15.061367] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E50-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1736326275056643000, start_id=1736326275056643000, end_id=1736326275056643001) [2025-01-08 00:51:15.066422] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=103] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8649, offset=2454671}, end_cursor:ObLogCursor{file_id=1, log_id=8650, offset=2454968}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a18050, len:2929, log_data_len:297, seq:8649, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:15.066826] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E50-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:15.066910] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E50-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326274729532001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326274729532001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:15.075070] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=57] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8650, offset=2454968}, end_cursor:ObLogCursor{file_id=1, log_id=8651, offset=2455265}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a1c050, len:2632, log_data_len:297, seq:8650, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:15.076300] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=49] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8651, offset=2455265}, end_cursor:ObLogCursor{file_id=1, log_id=8652, offset=2455562}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a20050, len:2335, log_data_len:297, seq:8651, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:15.095690] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A3-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:15.095744] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A3-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326275013027000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326275013027000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:15.105121] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=51] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:15.113794] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=47] ====== check clog disk timer task ====== [2025-01-08 00:51:15.113831] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=32] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:15.122655] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=28] ts response task statistics(alloc_count=59978, free_count=29988) [2025-01-08 00:51:15.135063] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=75] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:15.144150] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.152518] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:15.175577] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=89] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:15.244296] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=168][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.255642] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A76-0-0] [lt=147] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326275255607}) [2025-01-08 00:51:15.255743] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285977-0-0] [lt=23] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:15.256391] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285977-0-0] [lt=25] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275256367, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275256367, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275256367, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:15.256440] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285977-0-0] [lt=48] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326275255607}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275256367, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275256367, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275256367, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:15.257924] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=35] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:15.262168] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=36] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1003}, {id:1}]) [2025-01-08 00:51:15.267602] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:15.269322] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=43] gc stale ls task succ [2025-01-08 00:51:15.269743] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=27] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:15.269770] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=24] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:15.269782] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:15.269793] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:15.272596] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:51:15.273422] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=822] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:15.274355] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-01-08 00:51:15.274382] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:15.275436] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=30] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:15.276322] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:15.276407] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=77] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:15.276424] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=12] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:15.276446] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=21] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:15.282664] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853B4-0-0] [lt=25] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:15.282708] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853B4-0-0] [lt=36] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:15.283838] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=44] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:15.286842] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=29] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:15.288694] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=23] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:15.288714] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=19] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:15.290450] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:15.290727] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084879-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:15.290793] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084879-0-0] [lt=19] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:15.290808] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084879-0-0] [lt=12] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=3, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:15.293847] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=14] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:15.298650] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30] start to print blacklist info [2025-01-08 00:51:15.299006] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=25] ls blacklist refresh finish(cost_time=45032) [2025-01-08 00:51:15.301023] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=20] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:15.302612] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=26] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:15.302631] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=18] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:15.304020] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084879-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:15.304090] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084879-0-0] [lt=18] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:15.304106] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084879-0-0] [lt=12] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:15.305421] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:15.307502] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=13] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:15.308314] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3B0-0-0] [lt=40][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:15.309172] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=18] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:15.310599] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=18] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:15.310617] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=18] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:15.312047] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:15.312163] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:15.313348] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084879-0-0] [lt=7] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:15.313423] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084879-0-0] [lt=25] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:15.313437] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084879-0-0] [lt=12] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:15.313458] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=11] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:15.314044] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=582] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:15.314097] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=44] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:15.314108] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:15.314129] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084879-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:15.314148] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084879-0-0] [lt=18] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:15.314143] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:15.314158] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=6] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:15.314165] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:15.314181] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084879-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:15.314192] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084879-0-0] [lt=11] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:15.314201] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=5] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:15.314259] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=57] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:15.314281] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084879-0-0] [lt=18] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:15.314296] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084879-0-0] [lt=14] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:15.314300] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%", tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-01-08 00:51:15.314371] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=47] gc stale ls task succ [2025-01-08 00:51:15.315590] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:15.315610] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=18] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:15.315620] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=8] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:15.315655] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:15.315668] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:15.315676] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:15.315688] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:15.315704] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084879-0-0] [lt=14] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:15.319077] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.319731] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.320285] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=85] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:15.322264] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=117][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:15.322364] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%", tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:15.335842] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:15.337901] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=18] Garbage Collector is running(seq_=378, gc_interval=10000000) [2025-01-08 00:51:15.337952] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=24] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:15.337973] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] gc_check_member_list_ cost time(ret=0, time_us=47) [2025-01-08 00:51:15.337986] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:51:15.337997] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:15.338010] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:15.338020] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=11) [2025-01-08 00:51:15.338027] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:51:15.339381] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:15.342738] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=6] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:15.343392] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=38] PNIO [ratelimit] time: 1736326275343390, bytes: 873771722, bw: 0.303320 MB/s, add_ts: 1003873, add_bytes: 319286 [2025-01-08 00:51:15.344422] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.347254] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB1-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:15.347278] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB1-0-0] [lt=22] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:15.353332] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737416192, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:15.360796] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:15.360821] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:15.369830] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87097-0-0] [lt=14][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:15.369876] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:15.369888] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:15.373238] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=14] current monitor number(seq_=-1) [2025-01-08 00:51:15.378752] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=169] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:15.378820] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185426-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:15.392753] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:15.392979] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=52] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:15.393013] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:15.393029] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=96) [2025-01-08 00:51:15.403918] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] get_max_applied_scn(scn={val:1736326275122907001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2169017}, palf_committed_end_scn_:{val:1736326275122907002, v:0}, last_check_scn_:{val:1736326275122907002, v:0}, max_applied_cb_scn_:{val:1736326275122907001, v:0}}) [2025-01-08 00:51:15.403953] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2169017}, readable_log_tail:{lsn:2169017}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64935751, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:15.407280] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA685E-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4198) [2025-01-08 00:51:15.413914] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:15.413963] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=37] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:15.413980] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:15.413988] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:15.414040] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=50] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:15.414054] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=94) [2025-01-08 00:51:15.423623] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84303644}], scn_array=[cnt:1, {val:1736326275399918000, v:0}], curr_size=586, accum_size=3255, time_cost=20072) [2025-01-08 00:51:15.424619] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D04A-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:15.430157] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58460][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45BF-0-0] [lt=64] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:15.433404] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:15.433434] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:15.433452] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:15.433465] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:15.434562] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=230] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1643) [2025-01-08 00:51:15.434631] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2929, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:15.434652] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:15.434666] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=187, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:15.434704] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=68, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:15.437667] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:15.438530] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:15.438711] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58460][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=26] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:15.441822] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=59] [PALF STAT APPEND COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:15.443907] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85AF-0-0] [lt=32] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:15.443983] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85AF-0-0] [lt=75] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6240, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:15.444099] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:15.444599] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.459384] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=207] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.459494] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.461276] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:15.461321] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=37] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71333860, tenant_id=1001) [2025-01-08 00:51:15.461346] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:15.463290] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=30] ====== tenant freeze timer task ====== [2025-01-08 00:51:15.472302] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=57] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:15.472359] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=49] temporary file module use no blocks [2025-01-08 00:51:15.472372] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:15.472385] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:15.472398] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193779385931, flush_io_finished_ret_:0, flush_io_finished_round_:1934, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:15.482432] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=42] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:15.482477] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=39] temporary file module use no blocks [2025-01-08 00:51:15.482488] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:15.482501] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:15.482514] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193779386380, flush_io_finished_ret_:0, flush_io_finished_round_:3762, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:15.496105] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285978-0-0] [lt=27] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12274, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:15.496952] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=74] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:15.506623] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285978-0-0] [lt=21] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12274, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275506567, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275506567, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275506567, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:15.506933] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285978-0-0] [lt=307] success to process renew task(task={tenant_id:1, table_id:12274, add_timestamp:1736326275496008}, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275506567, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275506567, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326275506567, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:15.519350] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=32] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.533629] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:15.533730] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=96] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:15.534815] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2380, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:15.535012] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=196] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1333) [2025-01-08 00:51:15.535032] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=259, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:15.535089] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=41, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:15.537432] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB APPEND COST TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=124, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:15.537526] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=92] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=5365, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:15.537539] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=13] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:15.537547] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=-1312, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:15.537554] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=4178, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:15.539967] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:15.542781] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.544276] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=92] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.546905] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:15.546949] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.547327] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58473][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:15.547325] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:15.553739] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:15.561700] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4085) [2025-01-08 00:51:15.561741] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:15.567020] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:15.567404] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=126] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48935, [delay/us]:prepare:12, schedule:131, submit:275, rt:2774, total:3194) [2025-01-08 00:51:15.567445] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:15.567504] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48923, [delay/us]: prepare:12, schedule:131, submit:275, device:2774, total:3194) [2025-01-08 00:51:15.567518] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:15.567557] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4742, iops:19, bw:89692, [delay/us]:prepare:7, schedule:61, submit:89, rt:2083, total:2242) [2025-01-08 00:51:15.567597] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:15.567710] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=98] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4743, avg_iops:19, avg_bw:89683, [delay/us]: prepare:7, schedule:61, submit:89, device:2083, total:2242) [2025-01-08 00:51:15.567789] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:15.567817] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16304, [delay/us]:prepare:6, schedule:133, submit:1, rt:1029, total:1170) [2025-01-08 00:51:15.567862] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:163048, [delay/us]:prepare:14, schedule:256, submit:341, rt:1658, total:2271) [2025-01-08 00:51:15.567874] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:15.567906] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16304, [delay/us]: prepare:6, schedule:133, submit:1, device:1029, total:1170) [2025-01-08 00:51:15.567921] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:163043, [delay/us]: prepare:14, schedule:256, submit:341, device:1658, total:2271) [2025-01-08 00:51:15.567930] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:15.567973] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4076, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:349, total:398) [2025-01-08 00:51:15.568000] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:15.568061] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4075, [delay/us]: prepare:0, schedule:0, submit:600000000, device:349, total:398) [2025-01-08 00:51:15.568126] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:15.568164] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:15.568238] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=70] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:15.568268] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:15.568278] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=285kB/s, net_out=319kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:15.578674] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15022585576, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:15.578840] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=107] recycle one thread(this=0x2ad8881fc3b0, total=5, remain=4) [2025-01-08 00:51:15.578857] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15022585576, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:15.578868] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7788873459, tenant_id:1001}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:15.579013] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=66] recycle one thread(this=0x2ad8f90b0e70, total=6, remain=5) [2025-01-08 00:51:15.579033] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7788873459, tenant_id:1001}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:15.600711] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=43226669, this={dio_aligned_buf:{buf_write_offset:2170, buf_padding_size:1926, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:45564, truncate_used_ts:13399}, log_block_size:67108864, io_fd:{first_id:-1, second_id:160, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=43225088, buf_len=589, total_write_size=82887, total_write_size_after_dio=675840, ob_pwrite_used_ts=2191800, count=145, each_pwrite_cost=15115) [2025-01-08 00:51:15.603402] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58474][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:15.603537] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58474][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3965-0-0] [lt=51] init point thread id with(&point=0x5572d50e43c0, idx_=1130, point=[thread id=58474, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58474) [2025-01-08 00:51:15.605041] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1001, group_id_=1, token=3) [2025-01-08 00:51:15.623742] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=47] get number(ret=0, service_type_=0, range=1, base_id=1736326275620896000, start_id=1736326275620896000, end_id=1736326275620896001) [2025-01-08 00:51:15.623799] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326275623732], gts:1736326275620896000, latest_srr:[mts=1736326275623732]}) [2025-01-08 00:51:15.623828] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326275623818], gts:1736326275620896000, latest_srr:[mts=1736326275623818]}) [2025-01-08 00:51:15.624124] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326275624110], gts:1736326275620896000, latest_srr:[mts=1736326275624110]}) [2025-01-08 00:51:15.624650] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8A-0-0] [lt=24] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326275720174, keepalive_ts:-1}) [2025-01-08 00:51:15.624679] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8A-0-0] [lt=28] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326275720174, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:15.627469] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:195) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C9A-0-0] [lt=32][errcode=-4018] table_operator_ get_source failed(ret=-4018) [2025-01-08 00:51:15.628948] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C9A-0-0] [lt=9][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:15.630324] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2190254}], scn_array=[cnt:1, {val:1736326275267292003, v:0}], curr_size=122, accum_size=854, time_cost=12389) [2025-01-08 00:51:15.631105] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C9A-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:15.631458] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43223162}], scn_array=[cnt:1, {val:1736326275611769000, v:0}], curr_size=516, accum_size=8268, time_cost=13086) [2025-01-08 00:51:15.631472] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45C3-0-0] [lt=9] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:15.634186] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85B5-0-0] [lt=126] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:15.634276] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85B5-0-0] [lt=89] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21229, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:15.634291] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85B5-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=16, avg_log_submit_to_flush_time=2680, avg_log_submit_to_slide_time=21212) [2025-01-08 00:51:15.634635] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:15.634780] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=137] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18006, miss_cnt:0, cache_read_size:2202897, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:15.634950] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=221] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=142, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:15.634971] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:15.646994] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.669841] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853B6-0-0] [lt=10] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:15.669899] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853B6-0-0] [lt=49] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:15.680607] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C644-0-0] [lt=87][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193780594437, cache_obj->added_lc()=false, cache_obj->get_object_id()=13616, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:15.681869] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=35] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:15.681892] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=23] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:15.681981] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84983-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:15.682003] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84983-0-0] [lt=18] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:15.713455] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3867-0-0] [lt=25][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17995, log_task={header:{begin_lsn:{lsn:2190254}, end_lsn:{lsn:2190376}, log_id:17995, min_scn:{val:1736326275267292003, v:0}, max_scn:{val:1736326275267292003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2190132}, prev_proposal_id:1, committed_end_lsn:{lsn:2190254}, data_checksum:940009244, accum_checksum:2522156517, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326275612967, freeze_ts:1736326275612967, submit_ts:1736326275612973, flushed_ts:1736326275630682, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:17709}, fs_cb_begin_ts=1736326275713444, log_life_time=100477) [2025-01-08 00:51:15.714753] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085155-0-0] [lt=25] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:15.717332] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91EF-0-0] [lt=112] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:15.729813] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/74:172672 disk:308K/74:172672 t1m2:52K/12:114533 t1m2g0:52K/12:114533 t1002m2:168K/42:248302 t1002m2g0:168K/42:248302 t1001m2:88K/20:48734 t1001m2g0:88K/20:48734 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:15.731635] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45C5-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:15.733740] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=76, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:15.733769] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=2335, avg_group_log_size=233) [2025-01-08 00:51:15.735012] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2261, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:15.735053] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2335) [2025-01-08 00:51:15.735125] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=66] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=250, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:15.735156] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:15.738603] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91F0-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:15.738625] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91F0-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5513, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:15.740918] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:15.747169] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.754469] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:15.768555] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE85A-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326275768371000, start_id=1736326275768371000, end_id=1736326275768371001) [2025-01-08 00:51:15.772903] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:15.773394] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58466][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=51] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-01-08 00:51:15.779229] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5236929622, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:15.779359] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=49] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:15.779375] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5236929622, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:15.789952] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-01-08 00:51:15.789979] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:15.790680] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=110] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:15.799244] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AD20-0-0] [lt=28] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.863697242736816406e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=393, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.343607080527527842e+02) [2025-01-08 00:51:15.804420] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=55] PNIO [ratelimit] time: 1736326275804417, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005182, add_bytes: 0 [2025-01-08 00:51:15.805793] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=170] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=62, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:15.805826] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5876, avg_group_log_size=452) [2025-01-08 00:51:15.806376] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:15.807265] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=3276, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:15.807291] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5876) [2025-01-08 00:51:15.807309] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=926, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:15.807337] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:15.808731] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BCB-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:15.808755] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BCB-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=6681, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:15.808765] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BCB-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=853, avg_log_gen_to_submit_time=929, avg_log_submit_to_flush_time=4396, avg_log_submit_to_slide_time=5751) [2025-01-08 00:51:15.808847] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5876, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:15.812289] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=69] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:15.812339] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:15.813812] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3869-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:15.813861] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3869-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:15.820489] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=2079744) [2025-01-08 00:51:15.842299] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=86, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:15.843007] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:15.845778] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58463][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA396A-0-0] [lt=90] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10825) [2025-01-08 00:51:15.847383] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=100][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.847974] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326275641248000, v:0}, new_max_version={val:1736326275641248000, v:0}, last_min_version={val:1736326275234333000, v:0}, last_max_version={val:1736326275234333000, v:0}, total_time_us=7642, delta_us=206724) [2025-01-08 00:51:15.848051] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=73] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326275641248000, v:0}, min_version={val:1736326275641248000, v:0}, max_version={val:1736326275641248000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:15.875999] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:15.897247] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.897435] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.897466] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.897490] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:15.918406] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3019, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:15.918439] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:15.918460] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=729, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:15.918517] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=224] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:15.922112] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDE5-0-0] [lt=48][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193780835946, cache_obj->added_lc()=false, cache_obj->get_object_id()=13617, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:15.923534] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91F5-0-0] [lt=46] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=26, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:15.923579] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91F5-0-0] [lt=44] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21211, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:15.923593] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A91F5-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=17, avg_log_submit_to_flush_time=3583, avg_log_submit_to_slide_time=21194) [2025-01-08 00:51:15.923715] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:15.923773] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17908, miss_cnt:0, cache_read_size:2190742, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:15.941342] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:15.947466] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:15.954951] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:15.964489] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=10279, avg_group_log_size=541) [2025-01-08 00:51:15.964520] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10279) [2025-01-08 00:51:15.965852] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=2482, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:15.965880] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10279) [2025-01-08 00:51:15.965898] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=174, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:15.965926] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=33] [PALF STAT FLUSH CB COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=30, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:15.968007] INFO [LIB] stat (utility.h:1210) [58463][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA396B-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=14, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:15.968043] INFO [LIB] stat (utility.h:1210) [58463][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA396B-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=4399, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:15.968055] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58463][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA396B-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=95, avg_log_gen_to_submit_time=201, avg_log_submit_to_flush_time=2722, avg_log_submit_to_slide_time=4198) [2025-01-08 00:51:15.968454] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=365] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10279, hit_cnt=19, read_cnt=19, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:15.971652] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FA-0-0] [lt=39] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:15.971694] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FA-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326275768371001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326275768371001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:15.978312] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=194, gc_interval=10000000) [2025-01-08 00:51:15.978369] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=36] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:15.978391] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=13] gc_check_member_list_ cost time(ret=0, time_us=42) [2025-01-08 00:51:15.978403] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:15.980023] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=6] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:15.980069] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=41] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:15.980085] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=14] execute_gc cost time(ret=0, time_us=16) [2025-01-08 00:51:15.980094] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyHandler start process [2025-01-08 00:51:15.983468] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5FA-0-0] [lt=13] get number(ret=0, service_type_=0, range=1, base_id=1736326275982428000, start_id=1736326275982428000, end_id=1736326275982428001) [2025-01-08 00:51:15.986875] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=21] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:16.003690] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326275801793000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=201888, in_cluster_service=true, cluster_version={val:1736326275601562000, v:0}, min_cluster_version={val:1736326275601562000, v:0}, max_cluster_version={val:1736326275601562000, v:0}, get_cluster_version_err=0, cluster_version_delta=402119, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36483, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=530500668, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:16.010514] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=69] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:16.011085] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58468][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6865-0-0] [lt=39] init point thread id with(&point=0x5572d50e4240, idx_=1124, point=[thread id=58468, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58468) [2025-01-08 00:51:16.037055] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85C1-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=138, avg_log_submit_to_flush_time=2552, avg_log_submit_to_slide_time=4684) [2025-01-08 00:51:16.037167] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=176] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2034, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:16.040878] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] no need to update min/max version(new_version={val:1736326275641248000, v:0}, min_version={val:1736326275641248000, v:0}, max_version={val:1736326275641248000, v:0}) [2025-01-08 00:51:16.048796] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.081847] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8652, offset=2455562}, end_cursor:ObLogCursor{file_id=1, log_id=8653, offset=2455867}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a24050, len:2038, log_data_len:305, seq:8652, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:16.082244] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E51-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:16.082384] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E51-0-0] [lt=119] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326275620896002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326275620896002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:16.092838] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:16.092855] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=58] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8653, offset=2455867}, end_cursor:ObLogCursor{file_id=1, log_id=8654, offset=2456164}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a28050, len:1733, log_data_len:297, seq:8653, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:16.093491] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E51-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:16.093549] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E51-0-0] [lt=44] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326275620896003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326275620896003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:16.105394] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=36] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:16.107574] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8654, offset=2456164}, end_cursor:ObLogCursor{file_id=1, log_id=8655, offset=2456461}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a2c050, len:1436, log_data_len:297, seq:8654, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:16.110542] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8655, offset=2456461}, end_cursor:ObLogCursor{file_id=1, log_id=8656, offset=2456758}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a30050, len:1139, log_data_len:297, seq:8655, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:16.124516] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=613, get_gts_with_stc_cnt=39, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=37, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:16.124553] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:16.124653] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=525, get_gts_with_stc_cnt=76, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=74, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:16.129407] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A4-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:16.129456] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A4-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326276013031000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326276013031000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:16.143033] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=56] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1242512, replace_num=15728) [2025-01-08 00:51:16.143082] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=48] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:16.149372] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:16.149509] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=134][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.154170] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB028-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1736326276153097000, start_id=1736326276153097000, end_id=1736326276153097001) [2025-01-08 00:51:16.155564] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:16.157107] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=32] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=754968, clean_num=31457) [2025-01-08 00:51:16.162128] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB028-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326276161253000, start_id=1736326276161253000, end_id=1736326276161253001) [2025-01-08 00:51:16.163357] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1003, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18000, max_lsn:2190986, max_scn:{val:1736326275768371003, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2190986}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2190864}, last_submit_end_lsn:{lsn:2190986}, last_submit_log_id:18000, last_submit_log_pid:1, max_flushed_lsn:{lsn:2190864}, max_flushed_end_lsn:{lsn:2190986}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2190864}, last_slide_log_id:17999, last_slide_scn:{val:1736326275768371002, v:0}, last_slide_lsn:{lsn:2190742}, last_slide_end_lsn:{lsn:2190864}, last_slide_log_pid:1, last_slide_log_accum_checksum:3400925526, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924fd6530}, [StateMgr]={this:0x2ad924fda830, self:"192.168.159.137:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.139:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18000, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad924fe12f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326273652609, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326273652609, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924fd7c70}, [ModeMgr]={palf_id:1003, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436825152001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/meta", dir_fd:181, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924fdca38, flashback_version:0}, log_storage:{log_tail:{lsn:2190986}, readable_log_tail:{lsn:2190986}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64913782, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, palf_epoch:4, last_purge_throttling_ts:-1, this:0x2ad924fdaf70}, [Reconfirm]={palf_id:1003, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924fda9b0}) [2025-01-08 00:51:16.163814] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=450][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:16.163833] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:16.163845] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:16.163854] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] current log_task status(palf_id=1003, self="192.168.159.137:2882", log_id=18000, log_task={header:{begin_lsn:{lsn:2190864}, end_lsn:{lsn:2190986}, log_id:18000, min_scn:{val:1736326275768371003, v:0}, max_scn:{val:1736326275768371003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2190742}, prev_proposal_id:1, committed_end_lsn:{lsn:2190864}, data_checksum:940009244, accum_checksum:4090651506, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1736326276116079, freeze_ts:1736326276116079, submit_ts:1736326276116091, flushed_ts:1736326276118849, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:2758}, ack_info_list=[]) [2025-01-08 00:51:16.172531] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B5BBA6D-0-0] [lt=63] get number(ret=0, service_type_=0, range=1, base_id=1736326276171044000, start_id=1736326276171044000, end_id=1736326276171044001) [2025-01-08 00:51:16.178381] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58476][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:16.178575] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread created(tenant_->id()=1001, group_id_=1, token=4) [2025-01-08 00:51:16.204335] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58476][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2CDA-0-0] [lt=29] init point thread id with(&point=0x5572d50e4440, idx_=1132, point=[thread id=58476, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58476) [2025-01-08 00:51:16.223372] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [46346][T1_ArcFetcher][T1][YB42C0A89F89-00062B2D1678458E-0-0] [lt=24] ObArchiveFetcher is running(thread_index=0) [2025-01-08 00:51:16.233463] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45CF-0-0] [lt=33][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18103, log_task={header:{begin_lsn:{lsn:2203385}, end_lsn:{lsn:2203507}, log_id:18103, min_scn:{val:1736326276013031001, v:0}, max_scn:{val:1736326276013031001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2203263}, prev_proposal_id:1, committed_end_lsn:{lsn:2203385}, data_checksum:1715074492, accum_checksum:4271381875, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326276132935, freeze_ts:1736326276132935, submit_ts:1736326276132948, flushed_ts:1736326276135893, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:2945}, fs_cb_begin_ts=1736326276233449, log_life_time=100514) [2025-01-08 00:51:16.249963] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.258257] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=77] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:16.262465] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=85] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1002}, {id:1}]) [2025-01-08 00:51:16.267766] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=67] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:16.269474] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:16.272106] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3B3-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:16.289822] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C9BB5D3-0-0] [lt=51] get number(ret=0, service_type_=0, range=1, base_id=1736326276287388000, start_id=1736326276287388000, end_id=1736326276287388001) [2025-01-08 00:51:16.291479] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D170BBDA3-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326276291359000, start_id=1736326276291359000, end_id=1736326276291359001) [2025-01-08 00:51:16.291962] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=45] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:16.302574] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD9-0-0] [lt=33] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:16.302620] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CD9-0-0] [lt=36] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=50) [2025-01-08 00:51:16.306791] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=37] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:16.312475] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=119] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:16.312690] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58406][T1001_TimerWK4][T1001][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:16.315085] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:51:16.315192] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3873-0-0] [lt=171][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18001, log_task={header:{begin_lsn:{lsn:2190986}, end_lsn:{lsn:2191108}, log_id:18001, min_scn:{val:1736326275768371004, v:0}, max_scn:{val:1736326275768371004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2190864}, prev_proposal_id:1, committed_end_lsn:{lsn:2190986}, data_checksum:940009244, accum_checksum:417108857, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326276215036, freeze_ts:1736326276215036, submit_ts:1736326276215047, flushed_ts:1736326276219548, gen_to_freeze cost time:0, gen_to_submit cost time:11, submit_to_flush cost time:4501}, fs_cb_begin_ts=1736326276315161, log_life_time=100125) [2025-01-08 00:51:16.315980] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:16.316247] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=259] temporary file module use no blocks [2025-01-08 00:51:16.316263] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:16.316279] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:16.316300] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193780229163, flush_io_finished_ret_:0, flush_io_finished_round_:1939, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:16.318938] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.319717] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.320814] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:16.337382] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:16.343237] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:16.343891] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=113] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:16.344168] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=43] PNIO [ratelimit] time: 1736326276344165, bytes: 874135103, bw: 0.346279 MB/s, add_ts: 1000775, add_bytes: 363381 [2025-01-08 00:51:16.349807] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB2-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:16.349841] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB2-0-0] [lt=31] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:16.350231] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.361373] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:16.361413] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:16.362615] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:16.365155] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8709D-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:16.365286] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:16.365353] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=64][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:16.373397] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=22] current monitor number(seq_=-1) [2025-01-08 00:51:16.378946] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=67] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:16.381085] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185427-0-0] [lt=37][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:16.408606] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=5167) [2025-01-08 00:51:16.416686] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58476][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2B-0-0] [lt=149] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326276418217, keepalive_ts:-1}) [2025-01-08 00:51:16.416722] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58476][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2B-0-0] [lt=35] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326276418217, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:16.421015] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D04D-0-0] [lt=82][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:16.432538] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45D1-0-0] [lt=116] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:16.435791] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=78, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:16.435894] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:16.435915] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1098) [2025-01-08 00:51:16.435929] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:16.437779] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=1667, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:16.437828] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:16.437931] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=93] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=231, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:16.437979] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=148] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=36, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:16.438112] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:51:16.438339] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85CB-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2335) [2025-01-08 00:51:16.450313] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.459352] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=88] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.459393] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.472573] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=47] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:16.472629] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=48] temporary file module use no blocks [2025-01-08 00:51:16.472642] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:16.472656] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:16.472669] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193780386291, flush_io_finished_ret_:0, flush_io_finished_round_:1935, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:16.482632] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:16.482676] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=39] temporary file module use no blocks [2025-01-08 00:51:16.482687] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:16.482698] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:16.482710] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193780396384, flush_io_finished_ret_:0, flush_io_finished_round_:3763, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:16.484528] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F4-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326276487384, keepalive_ts:-1}) [2025-01-08 00:51:16.484582] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F4-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326276487384, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:16.494649] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B5-0-0] [lt=15] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326276497759, keepalive_ts:-1}) [2025-01-08 00:51:16.494699] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B5-0-0] [lt=49] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326276497759, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:16.495635] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B6-0-0] [lt=17] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326276497834, keepalive_ts:-1}) [2025-01-08 00:51:16.495687] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B6-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326276497834, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:16.518881] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=42] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.535873] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2335) [2025-01-08 00:51:16.535918] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:16.538848] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85CE-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:16.538870] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85CE-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3873, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:16.538922] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:16.541879] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.542383] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=210] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:16.543293] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58379][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=11] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:16.543857] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:16.543916] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:16.544417] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.550489] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=96][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.563030] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:16.563639] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5166) [2025-01-08 00:51:16.563683] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:16.572613] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:16.572676] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48883, [delay/us]:prepare:9, schedule:35, submit:68, rt:1208, total:1322) [2025-01-08 00:51:16.572693] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:16.572749] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48895, [delay/us]: prepare:9, schedule:35, submit:68, device:1208, total:1322) [2025-01-08 00:51:16.572763] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:16.572802] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:18, bw:81492, [delay/us]:prepare:10, schedule:91, submit:126, rt:2534, total:2762) [2025-01-08 00:51:16.572813] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:16.572850] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:18, avg_bw:81500, [delay/us]: prepare:10, schedule:91, submit:126, device:2534, total:2762) [2025-01-08 00:51:16.572861] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:16.572883] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16301, [delay/us]:prepare:6, schedule:134, submit:1, rt:1303, total:1445) [2025-01-08 00:51:16.572905] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4311, iops:38, bw:163014, [delay/us]:prepare:12, schedule:137, submit:82, rt:1497, total:1730) [2025-01-08 00:51:16.572914] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:16.572953] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16301, [delay/us]: prepare:6, schedule:134, submit:1, device:1303, total:1445) [2025-01-08 00:51:16.572970] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4312, avg_iops:38, avg_bw:163016, [delay/us]: prepare:12, schedule:137, submit:82, device:1497, total:1730) [2025-01-08 00:51:16.572980] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:16.573000] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4075, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:315, total:322) [2025-01-08 00:51:16.573024] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:16.573112] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=82] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4075, [delay/us]: prepare:0, schedule:0, submit:600000000, device:315, total:322) [2025-01-08 00:51:16.573131] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:16.573147] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:16.573156] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:16.573164] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:16.573172] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=318kB/s, net_out=247kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:16.580141] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7793869770, tenant_id:1001}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:16.580176] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:16.580262] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=84] try_wait return eagain(this=0x2ad8f90b0e70, thread=0x2ad9047efd10) [2025-01-08 00:51:16.580281] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7793869770, tenant_id:1001}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:16.603754] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=89] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:16.612591] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF51-0-0] [lt=9] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326276705014, keepalive_ts:-1}) [2025-01-08 00:51:16.612644] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF51-0-0] [lt=60] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326276705014, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:16.614342] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB195-0-0] [lt=16] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:16.614380] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB195-0-0] [lt=31] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:16.614409] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB195-0-0] [lt=26] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:16.624912] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [58213][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1EBBB197-0-0] [lt=99] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326276622515, can_tenant_elr:true}) [2025-01-08 00:51:16.634942] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1405, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:16.634977] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:16.634997] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=142, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:16.635150] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=105] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:16.638059] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85D0-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=16, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:16.638142] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85D0-0-0] [lt=81] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=14990, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:16.640764] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C9B-0-0] [lt=183][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:16.642410] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=18] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=102, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:16.642450] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=39] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=4, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:16.644669] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C9B-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:16.650761] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.666242] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326272645954, last_first_register_time_us=1736324449183670, register_time_us=1736326272645954, curr_time_us=1736326276666238) [2025-01-08 00:51:16.666329] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326276666280, last_submit_register_req_time_us=1736326276666280, register_parent_reason=1) [2025-01-08 00:51:16.666349] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:16.699904] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C646-0-0] [lt=192][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193781613733, cache_obj->added_lc()=false, cache_obj->get_object_id()=13618, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:16.723766] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC387B-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:16.732560] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45D7-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:16.732601] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58469][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45D7-0-0] [lt=32] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:16.734642] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58470][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D7E1-0-0] [lt=38] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=22, avg_log_submit_to_flush_time=1576, avg_log_submit_to_slide_time=13677) [2025-01-08 00:51:16.734787] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:16.734820] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18016, miss_cnt:0, cache_read_size:2204117, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:16.735992] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=87, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:16.736017] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:51:16.736276] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:312K/71:116057 disk:312K/71:116057 t1m2:44K/10:22968 t1m2g0:44K/10:22968 t1002m2:192K/45:142514 t1002m2g0:192K/45:142514 t1001m2:76K/16:99830 t1001m2g0:76K/16:99830 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:16.737382] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1830, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:16.737501] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=116] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:16.737519] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=285, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:16.737644] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=68] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:16.738620] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85D4-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:16.738643] INFO [LIB] stat (utility.h:1210) [58469][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85D4-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3329, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:16.744997] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:16.747475] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326276746530000, start_id=1736326276746530000, end_id=1736326276746530001) [2025-01-08 00:51:16.750835] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.751249] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=27] ====== tenant freeze timer task ====== [2025-01-08 00:51:16.763602] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:16.769945] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D3D-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:16.773495] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50506][T1002_L0_G0][T1002][YB42C0A89F8A-00062B2D2299B9E7-0-0] [lt=56] get number(ret=0, service_type_=0, range=1, base_id=1736326276771269000, start_id=1736326276771269000, end_id=1736326276771269001) [2025-01-08 00:51:16.775520] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:16.775588] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84310104}], scn_array=[cnt:1, {val:1736326276622515000, v:0}], curr_size=122, accum_size=3375, time_cost=11540) [2025-01-08 00:51:16.775735] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58437][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=33] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-01-08 00:51:16.780448] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5238877755, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:16.780496] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:16.780509] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a1eb0) [2025-01-08 00:51:16.780518] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5238877755, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:16.792133] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:16.792442] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=91] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:16.804482] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=56] get_min_unreplayed_log_info(lsn={lsn:2204117}, scn={val:1736326276513112003, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326276734635, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2204117}, next_to_submit_scn_:{val:1736326276513112003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2203995}, end_lsn:{lsn:2204117}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2204117}, readable_log_tail:{lsn:2204117}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64900651, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326276513112002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326276513112002, v:0}, data_checksum:3348837161, flag:15811}}, init_mode_version:0, accumulate_checksum:4152384083, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326276513112002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18016, miss_cnt:0, cache_read_size:2204117, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:16.805949] INFO pn_ratelimit (group.c:653) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=18] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:16.807025] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:16.808795] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=69] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=647712768, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=426029056, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:16.808870] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=41] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:16.808890] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2835, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:16.808939] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:16.808987] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:16.809001] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:16.811635] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690E66-0-0] [lt=69] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.789484863281250000e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=894, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.777410179565279691e+01) [2025-01-08 00:51:16.811975] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=97, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:16.811999] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3961, avg_group_log_size=396) [2025-01-08 00:51:16.812023] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=66] PNIO [ratelimit] time: 1736326276812020, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007603, add_bytes: 0 [2025-01-08 00:51:16.812749] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:16.812919] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:16.816996] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58478][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:16.817267] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=102] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2791, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:16.817310] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3961) [2025-01-08 00:51:16.817343] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=211, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:16.817474] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:16.817497] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4821, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:16.817507] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=3188, avg_log_submit_to_slide_time=4704) [2025-01-08 00:51:16.817617] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=34, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:16.818059] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=170] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3961, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:16.818519] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-01-08 00:51:16.821047] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:16.824375] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC387D-0-0] [lt=25] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:16.824461] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC387D-0-0] [lt=65] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:16.829874] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=107] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:16.829904] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:16.829919] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=173) [2025-01-08 00:51:16.830021] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:16.830060] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=21] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=179) [2025-01-08 00:51:16.844157] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:16.850983] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=72][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.865449] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [46323][T1_RFLWorker][T1][YB42C0A89F89-00062B2D1618458E-0-0] [lt=37] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:51:16.896968] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.897019] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.897044] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.897068] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:16.898736] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=188] get_max_applied_scn(scn={val:1736326276771269000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18678536}, palf_committed_end_scn_:{val:1736326276771269000, v:0}, last_check_scn_:{val:1736326276771269000, v:0}, max_applied_cb_scn_:{val:1736326276771269000, v:0}}) [2025-01-08 00:51:16.898849] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=106] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18678536}, readable_log_tail:{lsn:18678536}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48426232, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:16.898956] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=81] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:16.926624] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1801, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:16.926714] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:16.926794] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=258, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:16.926974] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=83, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:16.927944] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9212-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=24, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:16.927990] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9212-0-0] [lt=44] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=32459, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:16.928004] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9212-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=18, avg_log_submit_to_flush_time=2159, avg_log_submit_to_slide_time=32440) [2025-01-08 00:51:16.928093] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:16.928428] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=324] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17918, miss_cnt:0, cache_read_size:2191962, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:16.945175] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=75, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:16.945397] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=60] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:16.946533] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9943) [2025-01-08 00:51:16.948857] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=76] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326276742307001, v:0}, new_max_version={val:1736326276742307001, v:0}, last_min_version={val:1736326276399807000, v:0}, last_max_version={val:1736326276399807000, v:0}, total_time_us=5583, delta_us=206547) [2025-01-08 00:51:16.948965] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=103] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326276742307001, v:0}, min_version={val:1736326276742307001, v:0}, max_version={val:1736326276742307001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:16.951410] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:16.964150] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:16.966247] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=33] LogLoopThread round_cost_time(us)(round_cost_time=16) [2025-01-08 00:51:16.996386] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FB-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:16.996438] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FB-0-0] [lt=32] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326276773972001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326276773972001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:16.998840] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=58] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326276273072003, v:0}, base_version={val:1736326276171044002, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326276513112003, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326276998817, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326276513112002, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326277056434, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326276273072003, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326276902817, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:17.001537] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=172] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=10410, avg_group_log_size=650) [2025-01-08 00:51:17.001580] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9943) [2025-01-08 00:51:17.004603] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2386, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:17.004632] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10410) [2025-01-08 00:51:17.004720] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=269, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:17.004748] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=77] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=37, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:17.005284] INFO [LIB] stat (utility.h:1210) [58476][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA397B-0-0] [lt=58] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=17, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:17.005325] INFO [LIB] stat (utility.h:1210) [58476][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA397B-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=3721, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:17.005337] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58476][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA397B-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=91, avg_log_submit_to_flush_time=2752, avg_log_submit_to_slide_time=3630) [2025-01-08 00:51:17.005471] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10410, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:17.015622] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5FB-0-0] [lt=104] get number(ret=0, service_type_=0, range=1, base_id=1736326277013355000, start_id=1736326277013355000, end_id=1736326277013355001) [2025-01-08 00:51:17.015844] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:149) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5FB-0-0] [lt=97] transaction rpc statistics(total_rt=7288, total_process=27, avg_rt=260) [2025-01-08 00:51:17.016320] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F4FE-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1736326277013355000, start_id=1736326277013355000, end_id=1736326277013355001) [2025-01-08 00:51:17.020397] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=40] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:17.039368] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9216-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=113, avg_log_submit_to_flush_time=2164, avg_log_submit_to_slide_time=3261) [2025-01-08 00:51:17.039576] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2066, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:17.051413] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=101][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.051995] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B9F-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:17.080451] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:17.080530] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=60] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=94) [2025-01-08 00:51:17.084862] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=27] Garbage Collector is running(seq_=195, gc_interval=10000000) [2025-01-08 00:51:17.084955] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=60] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:17.084974] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=15] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:17.086495] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:51:17.088473] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=95] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:51:17.088520] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=40] gc_check_member_list_ cost time(ret=0, time_us=3596) [2025-01-08 00:51:17.088543] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=13] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:51:17.090682] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:17.092412] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=49] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:17.093740] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=41] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:17.095129] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=25] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:17.095250] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=114] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:17.095271] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=20] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:17.095283] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:17.095294] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:17.095305] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=56) [2025-01-08 00:51:17.095314] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:51:17.095321] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-01-08 00:51:17.095337] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-01-08 00:51:17.101131] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84312065}, readable_log_tail:{lsn:84312065}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49897471, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:17.101398] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=264] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:17.101474] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=16] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:17.105547] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:17.114142] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=55] ====== check clog disk timer task ====== [2025-01-08 00:51:17.114173] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=26] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:17.121323] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8656, offset=2456758}, end_cursor:ObLogCursor{file_id=1, log_id=8657, offset=2457063}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a34050, len:842, log_data_len:305, seq:8656, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:17.121973] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E52-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:17.122137] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E52-0-0] [lt=83] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326276771269002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326276771269002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:17.133485] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=55] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8657, offset=2457063}, end_cursor:ObLogCursor{file_id=1, log_id=8658, offset=2457360}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a38050, len:537, log_data_len:297, seq:8657, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:17.134792] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E52-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:17.134834] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E52-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326276771269002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326276771269002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:17.139540] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=17] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:17.141627] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=42] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:17.146297] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=107] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:17.146527] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=47] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8658, offset=2457360}, end_cursor:ObLogCursor{file_id=1, log_id=8659, offset=2457657}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a3c050, len:4336, log_data_len:297, seq:8658, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:17.149355] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8659, offset=2457657}, end_cursor:ObLogCursor{file_id=1, log_id=8660, offset=2457954}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a40050, len:4039, log_data_len:297, seq:8659, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:17.151544] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.164778] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:17.183104] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A5-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:17.183155] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A5-0-0] [lt=34] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326277013355001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326277013355001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:17.183680] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:17.183798] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58476][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=11] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:51:17.221647] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18579E-0-0] [lt=89][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193782135478, cache_obj->added_lc()=false, cache_obj->get_object_id()=13619, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:17.222100] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=32] ====== traversal_flush timer task ====== [2025-01-08 00:51:17.222262] INFO [MDS] flush (mds_table_mgr.cpp:195) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485373-0-0] [lt=36] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326277123056000, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326290087570000, latest_log_ts:{val:1736326273510997001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.222390] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485373-0-0] [lt=94] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326272117597001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.222471] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485373-0-0] [lt=55] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326272010447001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.222612] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485373-0-0] [lt=110] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326272264082000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.222671] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485373-0-0] [lt=56] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:17.223604] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18579F-0-0] [lt=44][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:17.225187] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3885-0-0] [lt=158][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18010, log_task={header:{begin_lsn:{lsn:2192084}, end_lsn:{lsn:2192206}, log_id:18010, min_scn:{val:1736326276773972003, v:0}, max_scn:{val:1736326276773972003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2191962}, prev_proposal_id:1, committed_end_lsn:{lsn:2192084}, data_checksum:940009244, accum_checksum:3153134469, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326277124769, freeze_ts:1736326277124769, submit_ts:1736326277124832, flushed_ts:1736326277130434, gen_to_freeze cost time:0, gen_to_submit cost time:63, submit_to_flush cost time:5602}, fs_cb_begin_ts=1736326277225109, log_life_time=100340) [2025-01-08 00:51:17.227327] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857A1-0-0] [lt=83][errcode=-4201] hashtable not init [2025-01-08 00:51:17.233536] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2208701, this={dio_aligned_buf:{buf_write_offset:1079, buf_padding_size:3017, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:26417, truncate_used_ts:5447}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2207744, buf_len=122, total_write_size=9976, total_write_size_after_dio=327680, ob_pwrite_used_ts=2121221, count=77, each_pwrite_cost=27548) [2025-01-08 00:51:17.251760] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.258460] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:17.262647] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}, {id:1003}, {id:1002}, {id:1001}]) [2025-01-08 00:51:17.268033] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=79] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:17.269641] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3B6-0-0] [lt=47][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:17.269895] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:51:17.274511] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-01-08 00:51:17.274553] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=36] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:17.292339] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=73] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:17.307855] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:17.312933] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=17] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:17.313253] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=102] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:17.315243] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=48] gc stale ls task succ [2025-01-08 00:51:17.319022] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.319195] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.321281] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=59] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:17.333872] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=62] ====== checkpoint timer task ====== [2025-01-08 00:51:17.335019] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=47] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:17.335094] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=74] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:17.335170] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=29] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326290087570000, latest_log_ts:{val:1736326273510997001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.335327] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=153] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:17.335351] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:17.335496] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=141] get rec log scn(service_type_=0, rec_log_ts={val:1736326273510997001, v:0}) [2025-01-08 00:51:17.335508] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:17.335523] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:17.335543] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:17.335581] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:17.335606] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:17.335853] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=244] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:17.337056] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=151] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:17.337117] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=60] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:17.337281] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=139] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326272117597001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.337369] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=85] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:17.337386] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:17.337401] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:17.337415] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:17.337429] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2171213}, scn={val:1736326277123056001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:17.339949] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8660, offset=2457954}, end_cursor:ObLogCursor{file_id=1, log_id=8661, offset=2458251}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a44050, len:3742, log_data_len:297, seq:8660, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:17.340055] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=83] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326277123056001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:17.340087] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:17.340122] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=33] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:17.341408] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:17.341472] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=64] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:17.341518] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326272010447001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.341725] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=204] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:17.341741] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:17.341755] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:17.341770] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:17.341784] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2204849}, scn={val:1736326277013355003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:17.342305] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:17.342931] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1135) [2025-01-08 00:51:17.345254] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=301] PNIO [ratelimit] time: 1736326277345169, bytes: 874385846, bw: 0.238887 MB/s, add_ts: 1001004, add_bytes: 250743 [2025-01-08 00:51:17.346456] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:17.346517] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:17.349067] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8661, offset=2458251}, end_cursor:ObLogCursor{file_id=1, log_id=8662, offset=2458548}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a48050, len:3445, log_data_len:297, seq:8661, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:17.352110] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB3-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:17.352214] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB3-0-0] [lt=99] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:17.352453] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.353317] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=33] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326277013355003, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:17.353357] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:17.353380] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:17.353396] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:17.353416] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] get_min_unreplayed_log_info(lsn={lsn:2192450}, scn={val:1736326277272403001, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326277329612, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2192450}, next_to_submit_scn_:{val:1736326277272403001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2192328}, end_lsn:{lsn:2192450}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2192450}, readable_log_tail:{lsn:2192450}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64912318, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326277272403000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326277272403000, v:0}, data_checksum:2124447694, flag:21346}}, init_mode_version:0, accumulate_checksum:3870745369, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326277272403000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17922, miss_cnt:0, cache_read_size:2192450, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:17.354271] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=72] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:17.354307] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=35] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:17.354351] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326272264082000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.354417] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=63] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:17.354432] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:17.354448] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:17.354463] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:17.354478] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2192450}, scn={val:1736326277272403000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:17.354960] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=2151) [2025-01-08 00:51:17.355928] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=113] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8662, offset=2458548}, end_cursor:ObLogCursor{file_id=1, log_id=8663, offset=2458845}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a4c050, len:3148, log_data_len:297, seq:8662, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:17.356052] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326277272403000, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:17.356082] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:17.356101] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:17.356120] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:17.360983] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:17.361012] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:17.364400] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870A3-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:17.364444] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:17.364455] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:17.365432] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=57] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:17.379563] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=121] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:17.379908] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=28] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:17.381687] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=96] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:17.381804] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=41] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326274317657000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:17.381832] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326274317657000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326274317657000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:17.383324] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185428-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:17.389032] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:17.390286] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=47] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:17.390394] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=37] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326275719333000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:17.403632] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=62] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:17.413752] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get number(ret=0, service_type_=0, range=1, base_id=1736326277413698000, start_id=1736326277413698000, end_id=1736326277413698001) [2025-01-08 00:51:17.415625] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4084) [2025-01-08 00:51:17.417934] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58471][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6874-0-0] [lt=41] init point thread id with(&point=0x5572d50e4300, idx_=1127, point=[thread id=58471, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58471) [2025-01-08 00:51:17.417961] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D050-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:17.422226] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] ====== traversal_flush timer task ====== [2025-01-08 00:51:17.422380] INFO [MDS] flush (mds_table_mgr.cpp:195) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EDB-0-0] [lt=37] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326277345143000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289284252000, latest_log_ts:{val:1736326272625185001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.422552] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EDB-0-0] [lt=165] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:17.428701] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:17.430082] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=27] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:17.430174] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326275608151000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:17.430278] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=101] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326275608151000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326275608151000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:17.432683] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45E5-0-0] [lt=132] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:17.438247] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:17.439444] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:17.439476] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:17.439494] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:17.439507] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:17.442020] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2101, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:17.442048] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:17.442105] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1643) [2025-01-08 00:51:17.442125] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=274, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:17.442139] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:17.449644] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:17.449702] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=51] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:17.449718] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:17.449729] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:17.452700] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.452474] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=132, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}]) [2025-01-08 00:51:17.452842] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=364] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:17.457301] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:17.457331] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=23] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71333860, tenant_id=1001) [2025-01-08 00:51:17.457353] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:17.459355] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.459505] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.471640] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-01-08 00:51:17.475292] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=49] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:17.497565] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=34] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:17.519950] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.539376] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:17.539423] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=44] try gc retain ctx [2025-01-08 00:51:17.539433] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-01-08 00:51:17.539466] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326277123056003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326267132611003, v:0}, keep_alive_scn_:{val:1736326272117597004, v:0}, update_ts_:1736326277538889}) [2025-01-08 00:51:17.540148] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326290087570000, latest_log_ts:{val:1736326273510997001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:17.540431] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=163] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:17.540451] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:17.540478] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] finish update min start scn(max_decided_scn={val:1736326277123056003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326267132611003, v:0}, keep_alive_scn_:{val:1736326272117597004, v:0}, update_ts_:1736326277540007}) [2025-01-08 00:51:17.541506] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277123056001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:17.542064] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=123] finish update min start scn(max_decided_scn={val:1736326277514678000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326270015178001, v:0}, keep_alive_scn_:{val:1736326275013027002, v:0}, update_ts_:1736326277541099}) [2025-01-08 00:51:17.543417] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=60] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277013355003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:17.543501] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=72] finish update min start scn(max_decided_scn={val:1736326277272403002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326267890517000, v:0}, keep_alive_scn_:{val:1736326274264437001, v:0}, update_ts_:1736326277543414}) [2025-01-08 00:51:17.544266] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277272403000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326274729532002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:17.544397] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.545333] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.546563] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:17.546725] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:17.551441] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:17.551830] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58469][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=8] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:17.552879] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.553077] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9226-0-0] [lt=21] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:17.553099] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9226-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4518, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:17.553330] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:17.565952] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=17] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:17.566110] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4085) [2025-01-08 00:51:17.566132] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:17.576704] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:17.576753] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48952, [delay/us]:prepare:14, schedule:154, submit:796, rt:1728, total:2694) [2025-01-08 00:51:17.576768] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:17.576821] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48952, [delay/us]: prepare:14, schedule:154, submit:796, device:1728, total:2694) [2025-01-08 00:51:17.576834] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:17.576872] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4915, iops:15, bw:73429, [delay/us]:prepare:7, schedule:191, submit:346, rt:1310, total:1855) [2025-01-08 00:51:17.576882] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:17.576917] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4915, avg_iops:15, avg_bw:73429, [delay/us]: prepare:7, schedule:191, submit:346, device:1310, total:1855) [2025-01-08 00:51:17.576927] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:17.576948] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:7, bw:32635, [delay/us]:prepare:15, schedule:79, submit:0, rt:2347, total:2442) [2025-01-08 00:51:17.576970] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:171335, [delay/us]:prepare:7, schedule:262, submit:213, rt:1192, total:1675) [2025-01-08 00:51:17.576979] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12238, [delay/us]:prepare:1, schedule:38, submit:117, rt:317, total:474) [2025-01-08 00:51:17.576986] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:17.577014] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4681, avg_iops:7, avg_bw:32635, [delay/us]: prepare:15, schedule:79, submit:0, device:2347, total:2442) [2025-01-08 00:51:17.577029] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12238, [delay/us]: prepare:1, schedule:38, submit:117, device:317, total:474) [2025-01-08 00:51:17.577037] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:171335, [delay/us]: prepare:7, schedule:262, submit:213, device:1192, total:1675) [2025-01-08 00:51:17.577046] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:17.577065] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4079, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:341, total:350) [2025-01-08 00:51:17.577090] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:17.577116] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4079, [delay/us]: prepare:0, schedule:0, submit:600000000, device:341, total:350) [2025-01-08 00:51:17.577134] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:17.577148] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:17.577156] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:17.577267] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=106] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:17.577276] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=247kB/s, net_out=211kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:17.583253] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7797841904, tenant_id:1001}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:17.583467] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31] recycle one thread(this=0x2ad8f90b0e70, total=5, remain=4) [2025-01-08 00:51:17.583485] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] inc thread count(*this={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7797841904, tenant_id:1001}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:17.591974] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BECA-0-0] [lt=49] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:17.609129] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF52-0-0] [lt=40] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326277708003, keepalive_ts:-1}) [2025-01-08 00:51:17.609314] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF52-0-0] [lt=182] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326277708003, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:17.618922] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58478][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6876-0-0] [lt=28] init point thread id with(&point=0x5572d50e44c0, idx_=1134, point=[thread id=58478, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58478) [2025-01-08 00:51:17.630387] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:17.636505] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1396, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:17.636568] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=61] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:17.636594] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=207, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:17.636661] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=60, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:17.653038] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.654001] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C9C-0-0] [lt=104][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:17.656287] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C9C-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:17.657942] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BA1-0-0] [lt=84] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:17.673795] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326273652609, last_first_register_time_us=1736324454612308, register_time_us=1736326273652609, curr_time_us=1736326277673792) [2025-01-08 00:51:17.673884] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=35] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326277673832, last_submit_register_req_time_us=1736326277673832, register_parent_reason=1) [2025-01-08 00:51:17.673902] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:17.682708] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=34] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:17.682733] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:17.682768] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84984-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:17.682795] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84984-0-0] [lt=24] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:17.721068] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C648-0-0] [lt=41][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193782634899, cache_obj->added_lc()=false, cache_obj->get_object_id()=13620, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:17.725369] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC388F-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:17.732800] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=44, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:17.732844] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=43] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:17.734895] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45EB-0-0] [lt=86] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:17.734954] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45EB-0-0] [lt=47] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:17.739747] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=62] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=75, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:17.739777] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1634, avg_group_log_size=163) [2025-01-08 00:51:17.740012] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=48] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:320K/75:204216 disk:320K/75:204216 t1m2:48K/11:147197 t1m2g0:48K/11:147197 t1002m2:204K/49:232209 t1002m2g0:204K/49:232209 t1001m2:68K/15:154112 t1001m2g0:68K/15:154112 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:17.741962] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85F1-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:17.741989] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85F1-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3451, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:17.742000] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC85F1-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=18, avg_log_submit_to_flush_time=1715, avg_log_submit_to_slide_time=3569) [2025-01-08 00:51:17.742071] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:17.742104] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18026, miss_cnt:0, cache_read_size:2205337, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:17.742895] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2012, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:17.743043] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=144] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1634) [2025-01-08 00:51:17.743070] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=582, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:17.743155] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:17.743176] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4330, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:17.743194] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:17.747424] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:17.754014] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.766840] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:17.783719] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=55] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5240408898, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:17.783818] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:17.783834] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a08e0) [2025-01-08 00:51:17.783844] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5240408898, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:17.789629] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C64A-0-0] [lt=33][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193782703461, cache_obj->added_lc()=false, cache_obj->get_object_id()=13621, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:17.790126] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=248] ====== check clog disk timer task ====== [2025-01-08 00:51:17.790164] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=33] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:17.792549] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=72] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:17.797911] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C64F-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326277797248000, start_id=1736326277797248000, end_id=1736326277797248001) [2025-01-08 00:51:17.801605] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=6] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:17.806309] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:17.808260] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:17.813110] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:17.815647] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=97] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=71, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:17.815739] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=79] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=4650, avg_group_log_size=387) [2025-01-08 00:51:17.816780] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=117] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:17.817976] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=2697, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:17.818004] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4650) [2025-01-08 00:51:17.818027] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=158, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:17.818063] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=42, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:17.819151] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BE1-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=37, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:17.819272] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BE1-0-0] [lt=119] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=4009, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:17.819287] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BE1-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=98, avg_log_submit_to_flush_time=2908, avg_log_submit_to_slide_time=3911) [2025-01-08 00:51:17.819876] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4650, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:17.820011] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1736326277819968, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007948, add_bytes: 0 [2025-01-08 00:51:17.820756] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58489][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:17.820794] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-01-08 00:51:17.820845] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58489][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6879-0-0] [lt=60] init point thread id with(&point=0x5572d50e4780, idx_=1145, point=[thread id=58489, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58489) [2025-01-08 00:51:17.821525] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=2079744) [2025-01-08 00:51:17.825478] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3891-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:17.825532] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3891-0-0] [lt=44] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:17.835612] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=37] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:17.835653] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=29] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=107) [2025-01-08 00:51:17.850541] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:17.852416] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=34] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:17.854251] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.854637] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=91] ====== traversal_flush timer task ====== [2025-01-08 00:51:17.854825] INFO [MDS] flush (mds_table_mgr.cpp:195) [46305][T1_TimerWK1][T1][YB42C0A89F89-00062B2D17685755-0-0] [lt=73] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326277815176000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289254996000, latest_log_ts:{val:1736326272617414001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:17.854947] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46305][T1_TimerWK1][T1][YB42C0A89F89-00062B2D17685755-0-0] [lt=117] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:17.880593] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=8] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:51:17.889726] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=33] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:17.896794] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.896840] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.896866] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.896889] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:17.907148] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=29][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:17.908373] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=38][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:17.908426] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=17][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:17.922959] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58463][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC3-0-0] [lt=11] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326278022667, keepalive_ts:-1}) [2025-01-08 00:51:17.923007] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58463][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC3-0-0] [lt=45] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326278022667, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:17.925859] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58470][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3893-0-0] [lt=16][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18017, log_task={header:{begin_lsn:{lsn:2192938}, end_lsn:{lsn:2193060}, log_id:18017, min_scn:{val:1736326277773913000, v:0}, max_scn:{val:1736326277773913000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2192816}, prev_proposal_id:1, committed_end_lsn:{lsn:2192938}, data_checksum:940009244, accum_checksum:764310370, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326277825467, freeze_ts:1736326277825467, submit_ts:1736326277825524, flushed_ts:1736326277829102, gen_to_freeze cost time:0, gen_to_submit cost time:57, submit_to_flush cost time:3578}, fs_cb_begin_ts=1736326277925846, log_life_time=100379) [2025-01-08 00:51:17.927094] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1963, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:17.927133] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:17.927230] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=89] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=252, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:17.927311] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=69] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:17.928693] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D7F8-0-0] [lt=50] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:17.928729] INFO [LIB] stat (utility.h:1210) [58470][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D7F8-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=32698, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:17.928741] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58470][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D7F8-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=2285, avg_log_submit_to_slide_time=32676) [2025-01-08 00:51:17.928870] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:17.928903] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17928, miss_cnt:0, cache_read_size:2193182, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:17.947621] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:17.948157] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8ACB5-0-0] [lt=21] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.887849187850952148e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=394, Throttle Trigger(MB)=236, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.254635096447166234e+02) [2025-01-08 00:51:17.948642] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=67] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=76, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:17.949921] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9820) [2025-01-08 00:51:17.951918] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=44] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326277600983000, v:0}, new_max_version={val:1736326277600983000, v:0}, last_min_version={val:1736326277398785000, v:0}, last_max_version={val:1736326277398785000, v:0}, total_time_us=5429, delta_us=350932) [2025-01-08 00:51:17.951955] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326277600983000, v:0}, min_version={val:1736326277600983000, v:0}, max_version={val:1736326277600983000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:17.954427] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:17.967406] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:18.002636] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=117] get number(ret=0, service_type_=0, range=1, base_id=1736326278002045000, start_id=1736326278002045000, end_id=1736326278002045001) [2025-01-08 00:51:18.002889] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9820, avg_group_log_size=654) [2025-01-08 00:51:18.002912] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9820) [2025-01-08 00:51:18.004708] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=1894, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:18.004737] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9820) [2025-01-08 00:51:18.004756] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=570, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:18.004787] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=49] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:18.005777] INFO [LIB] stat (utility.h:1210) [58463][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA398A-0-0] [lt=70] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:18.005800] INFO [LIB] stat (utility.h:1210) [58463][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA398A-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=3643, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:18.005810] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58463][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA398A-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=101, avg_log_submit_to_flush_time=2517, avg_log_submit_to_slide_time=3542) [2025-01-08 00:51:18.006034] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9820, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:18.016757] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50915][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F504-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326278012752000, start_id=1736326278012752000, end_id=1736326278012752001) [2025-01-08 00:51:18.029265] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FC-0-0] [lt=103] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:18.029314] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FC-0-0] [lt=36] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326277773913002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326277773913002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:18.044036] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=94, avg_log_submit_to_flush_time=3434, avg_log_submit_to_slide_time=4930) [2025-01-08 00:51:18.044267] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58473][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9234-0-0] [lt=37] init point thread id with(&point=0x5572d50e4380, idx_=1129, point=[thread id=58473, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58473) [2025-01-08 00:51:18.044533] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:18.048365] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=35] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:18.054618] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=113][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.105860] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:18.109967] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] [COMPACTION DAG STATUS] (start_time=1736326268069646, end_time=1736326278109964, dag_status={}) [2025-01-08 00:51:18.110018] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=47] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:18.110034] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:18.110042] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:18.110050] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:18.110057] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:18.110064] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:18.110071] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:18.110078] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:18.110084] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:18.110266] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=179] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110282] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110291] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110299] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110307] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110315] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110322] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110404] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=81] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110417] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110425] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110433] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110441] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110448] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110456] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110463] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110470] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110477] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110489] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110502] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110514] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110526] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110550] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110561] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110569] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110576] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110584] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110591] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110598] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110670] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=72] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110683] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110691] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110699] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110706] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110714] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110721] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110728] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110735] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110742] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110752] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110765] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110773] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110796] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110807] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110815] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110823] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110830] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110837] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110850] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110858] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110879] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110890] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110898] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110906] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110913] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110921] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110929] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110937] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110944] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110952] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110960] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110968] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110975] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110982] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110989] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.110996] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.111003] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.111010] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.111018] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.111025] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.111032] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.111039] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.111046] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.111053] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:18.111061] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:18.111068] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:18.111074] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:18.111078] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:18.111134] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=55] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:18.111141] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:18.111146] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:18.111151] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:18.111156] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:18.111215] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=58] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:18.111223] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:18.131052] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=46] ts response task statistics(alloc_count=60050, free_count=30024) [2025-01-08 00:51:18.147836] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:18.154765] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=103][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.157335] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=53] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8663, offset=2458845}, end_cursor:ObLogCursor{file_id=1, log_id=8664, offset=2459150}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a50050, len:2851, log_data_len:305, seq:8663, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:18.158034] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E53-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:18.158079] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E53-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326277629937004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326277629937004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:18.166992] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8664, offset=2459150}, end_cursor:ObLogCursor{file_id=1, log_id=8665, offset=2459447}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a54050, len:2546, log_data_len:297, seq:8664, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:18.168960] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:18.171029] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=37] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=786425, clean_num=31457) [2025-01-08 00:51:18.171817] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E53-0-0] [lt=35] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:18.171870] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E53-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326277629937004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326277629937004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:18.177796] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=43] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:18.181161] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=39] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8665, offset=2459447}, end_cursor:ObLogCursor{file_id=1, log_id=8666, offset=2459744}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a58050, len:2249, log_data_len:297, seq:8665, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:18.182439] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=83] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8666, offset=2459744}, end_cursor:ObLogCursor{file_id=1, log_id=8667, offset=2460041}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a5c050, len:1952, log_data_len:297, seq:8666, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:18.188028] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:18.189604] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58463][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1001, workers_.get_size()=3) [2025-01-08 00:51:18.225644] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A6-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:18.225698] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A6-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326278012752001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326278012752001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:18.229435] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46100][TimerWK2][T0][YB42C0A89F89-00062B2D17E8543C-0-0] [lt=33] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:18.254925] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.256169] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A77-0-0] [lt=16] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326278256152}) [2025-01-08 00:51:18.257364] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285979-0-0] [lt=115] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326278257343, sql_renew_time:1736326278257343, is_mark_fail:false}]) [2025-01-08 00:51:18.257404] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285979-0-0] [lt=39] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326278256152}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326278257343, sql_renew_time:1736326278257343, is_mark_fail:false}]) [2025-01-08 00:51:18.260376] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=75] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:18.266734] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=103] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1003}, {id:1}, {id:1001}]) [2025-01-08 00:51:18.268311] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:18.270053] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:51:18.274026] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=97] ls blacklist refresh finish(cost_time=19883) [2025-01-08 00:51:18.286875] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3B9-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:18.292718] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=72] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:18.307867] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDA-0-0] [lt=30] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:18.307927] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDA-0-0] [lt=51] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=64) [2025-01-08 00:51:18.308755] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:18.313651] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=83] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:18.315494] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=26] gc stale ls task succ [2025-01-08 00:51:18.316681] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=47] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:18.316803] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=114] temporary file module use no blocks [2025-01-08 00:51:18.316818] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:18.316833] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:18.316848] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193782230314, flush_io_finished_ret_:0, flush_io_finished_round_:1941, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:18.316977] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:18.319046] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.319329] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.321871] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=81] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=2079744) [2025-01-08 00:51:18.343458] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:18.345863] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1736326278345859, bytes: 874629142, bw: 0.231865 MB/s, add_ts: 1000690, add_bytes: 243296 [2025-01-08 00:51:18.351114] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=32] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1258240, replace_num=15728) [2025-01-08 00:51:18.351153] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:18.353759] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EE6-0-0] [lt=26] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:18.353851] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:18.357018] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB4-0-0] [lt=17] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:18.357052] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB4-0-0] [lt=30] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:18.358406] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.363481] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:18.363507] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:18.364265] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:51:18.364382] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:51:18.365599] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=637538304, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:51:18.365629] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] === Run print tenant memory usage task === [2025-01-08 00:51:18.365726] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=9] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,612,800 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 637,538,304 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,278,364,586 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 160,381,740 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,042,788,352 max_mem_memstore_can_get_now= 801,908,736 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,278,364,586 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 71,333,860 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 645,615,616 max_mem_memstore_can_get_now= 356,669,398 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,278,364,586 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 240,672,760 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,114,390,528 max_mem_memstore_can_get_now= 1,203,363,818 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:51:18.365944] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=13] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,657,314,304 total_hold= 3,856,662,528 used= 3,471,478,784 freelists_hold= 185,835,520 total_maps= 1,632 total_unmaps= 240 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,612,800 virtual_memory_used= 4,829,216,768 [CHUNK_MGR] 2 MB_CACHE: hold= 83,886,080 count= 40 pushes= 1,821,117 pops= 1,821,077 maps= 1,501 unmaps= 235 [CHUNK_MGR] 4 MB_CACHE: hold= 7,745,536 count= 3 pushes= 6,154 pops= 6,151 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,100 pops= 3,096 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:51:18.365971] INFO print (ob_malloc_time_monitor.cpp:37) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=11] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 1118, delta_count= 1, avg_cost_time= 1118 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 71576, delta_count= 3, avg_cost_time= 23858 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 900800, delta_count= 3, avg_cost_time= 300266 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 1104682, delta_count= 1, avg_cost_time= 1104682 [2025-01-08 00:51:18.367873] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870A9-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:18.367918] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:18.367930] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:18.373580] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:18.373880] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=40] current monitor number(seq_=-1) [2025-01-08 00:51:18.379804] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=48] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:18.385934] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185429-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:18.417843] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D053-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:18.419987] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4105) [2025-01-08 00:51:18.433029] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58470][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45F9-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:18.438341] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:18.441097] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=37, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:18.441143] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:18.441238] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:18.441264] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:18.442496] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:18.443039] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2323, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:18.443080] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:18.443101] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1335, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:18.443243] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:18.458646] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.459468] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.459551] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.472934] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:18.472982] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=41] temporary file module use no blocks [2025-01-08 00:51:18.472994] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:18.473008] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:18.473020] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193782386720, flush_io_finished_ret_:0, flush_io_finished_round_:1937, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:18.482960] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:18.483044] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=78] temporary file module use no blocks [2025-01-08 00:51:18.483056] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:18.483068] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:18.483080] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193782396725, flush_io_finished_ret_:0, flush_io_finished_round_:3765, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:18.519858] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.542516] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:18.542544] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:18.542737] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.544309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.549727] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1736326278549553000, start_id=1736326278549553000, end_id=1736326278549553001) [2025-01-08 00:51:18.549913] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=50] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:18.551356] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:18.558812] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:18.558847] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.560839] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:18.561054] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58470][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=61] worker thread exit(tenant_->id()=1002, workers_.get_size()=6) [2025-01-08 00:51:18.568498] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4104) [2025-01-08 00:51:18.568528] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:18.574069] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6752137216, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:18.579238] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:18.579293] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:49027, [delay/us]:prepare:13, schedule:226, submit:47, rt:1702, total:1989) [2025-01-08 00:51:18.579310] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:18.579423] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=96] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:49024, [delay/us]: prepare:13, schedule:226, submit:47, device:1702, total:1989) [2025-01-08 00:51:18.579438] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:18.579479] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:14, bw:65365, [delay/us]:prepare:20, schedule:142, submit:131, rt:1489, total:1783) [2025-01-08 00:51:18.579541] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:18.579638] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:14, avg_bw:65361, [delay/us]: prepare:20, schedule:142, submit:131, device:1489, total:1783) [2025-01-08 00:51:18.579654] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:18.579679] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16339, [delay/us]:prepare:6, schedule:18, submit:0, rt:839, total:864) [2025-01-08 00:51:18.579741] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:163393, [delay/us]:prepare:6, schedule:144, submit:242, rt:1280, total:1674) [2025-01-08 00:51:18.579754] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:18.579786] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16338, [delay/us]: prepare:6, schedule:18, submit:0, device:839, total:864) [2025-01-08 00:51:18.579802] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:163388, [delay/us]: prepare:6, schedule:144, submit:242, device:1280, total:1674) [2025-01-08 00:51:18.579916] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:18.579943] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:369, total:378) [2025-01-08 00:51:18.580022] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=59] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:18.580108] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=80] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4083, [delay/us]: prepare:0, schedule:0, submit:600000000, device:369, total:378) [2025-01-08 00:51:18.580128] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:18.580144] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:18.580153] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:18.580161] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:18.580169] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=244kB/s, net_out=237kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:18.607346] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1736326278012752004, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3, epoch_tstamp:1736326278607289}, version_delta=594591) [2025-01-08 00:51:18.627689] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58490][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:18.628458] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68] worker thread created(tenant_->id()=1002, group_id_=1, token=6) [2025-01-08 00:51:18.631791] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=34] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326278631767], gts:1736326278631136000, latest_srr:[mts=1736326278631767]}) [2025-01-08 00:51:18.631823] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:18.631859] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326278631845], gts:1736326278631136000, latest_srr:[mts=1736326278631845]}) [2025-01-08 00:51:18.632258] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326278632243], gts:1736326278631136000, latest_srr:[mts=1736326278632243]}) [2025-01-08 00:51:18.635021] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45FD-0-0] [lt=52][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18126, log_task={header:{begin_lsn:{lsn:2206191}, end_lsn:{lsn:2206313}, log_id:18126, min_scn:{val:1736326278513691000, v:0}, max_scn:{val:1736326278513691000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2206069}, prev_proposal_id:1, committed_end_lsn:{lsn:2206191}, data_checksum:1715074492, accum_checksum:3787801850, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326278533105, freeze_ts:1736326278533105, submit_ts:1736326278533111, flushed_ts:1736326278534535, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:1424}, fs_cb_begin_ts=1736326278635011, log_life_time=101906) [2025-01-08 00:51:18.647237] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC860D-0-0] [lt=47] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:18.647267] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC860D-0-0] [lt=30] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4231, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:18.647353] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:18.648029] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43252836}, readable_log_tail:{lsn:43252836}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23851932, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:18.658965] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.665156] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C9D-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:18.666465] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C9D-0-0] [lt=92][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:18.720116] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=36] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:18.728048] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085157-0-0] [lt=23] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:18.728273] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38A3-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:18.733476] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=55] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=38, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:18.733516] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=39] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:18.735368] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45FF-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:18.735439] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A45FF-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:18.742649] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C650-0-0] [lt=52][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193783656478, cache_obj->added_lc()=false, cache_obj->get_object_id()=13622, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:18.743352] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2044, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:18.743381] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:18.743404] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=99, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:18.743565] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=65, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:18.744330] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=79, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:18.744443] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:18.744690] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=16] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:288K/69:111962 disk:288K/69:111962 t1m2:48K/11:209373 t1m2g0:48K/11:209373 t1002m2:176K/44:82500 t1002m2g0:176K/44:82500 t1001m2:64K/14:128018 t1001m2g0:64K/14:128018 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:18.745194] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=156] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:67, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2029659520, recv_hp_rpc_cnt:5229, recv_np_rpc_cnt:103576, recv_lp_rpc_cnt:0, recv_mysql_cnt:9333, recv_task_cnt:3735, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10796672, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=116010 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1837, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:93807, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:8, nesting_worker_cnt:0, token_change:1736326277819027}{group_id:10, queue_size:0, recv_req_cnt:11167, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2078, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326228853642}{group_id:2, queue_size:0, recv_req_cnt:30026, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11874, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:4966, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:26715, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=338 pcode=0x1502:cnt=287 pcode=0x717:cnt=187 pcode=0x52b:cnt=140 pcode=0x710:cnt=115, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:51:18.746297] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1102] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:15527117, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8186, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3572407, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=20, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:51:18.746899] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=601] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:14603300, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3559158, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:51:18.747558] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=657] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:19, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:874592961, recv_hp_rpc_cnt:3720, recv_np_rpc_cnt:37820, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:775941, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=65335 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:67645, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736326278188024}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2178, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326218958545}{group_id:2, queue_size:0, recv_req_cnt:15421, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18490, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=308 pcode=0x51c:cnt=232 pcode=0x717:cnt=173 pcode=0x710:cnt=92 pcode=0x150b:cnt=56, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:51:18.748257] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=698] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:58, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9511400696, recv_hp_rpc_cnt:77, recv_np_rpc_cnt:34814, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2934405, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=24964 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:0, recv_req_cnt:6991, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326269316514}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:149777, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:6, nesting_worker_cnt:0, token_change:1736326278627328}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326242277705}{group_id:19, queue_size:0, recv_req_cnt:406, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326264650105}{group_id:2, queue_size:0, recv_req_cnt:36981, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:11057, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=357 pcode=0x1501:cnt=184 pcode=0x1516:cnt=178 pcode=0x717:cnt=177 pcode=0x150b:cnt=113, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:51:18.751448] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=19] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:18.751665] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=101] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:18.753804] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2198, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:18.754021] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=168] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:18.754047] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=824, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:18.754746] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:18.754768] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4957, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:18.754781] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=35, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:18.756022] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB58-0-0] [lt=88] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:18.756055] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB58-0-0] [lt=27] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:18.759137] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.759784] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=79] clear dtl interm result cost(us)(clear_cost=3615, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:18.759909] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=9] BlockGCTimerTask success(ret=0, cost_time_us=18, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:0}}) [2025-01-08 00:51:18.760153] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=26] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:18.762279] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43253425}, readable_log_tail:{lsn:43253425}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23851343, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:18.762400] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=119] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:18.762438] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=15] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:18.767107] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=23] ====== tenant freeze timer task ====== [2025-01-08 00:51:18.769280] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] clear dtl interm result cost(us)(clear_cost=14, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:18.774604] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:18.777698] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D3E-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:18.779486] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE86D-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1736326278769992000, start_id=1736326278769992000, end_id=1736326278769992001) [2025-01-08 00:51:18.780339] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE86E-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326278780077000, start_id=1736326278780077000, end_id=1736326278780077001) [2025-01-08 00:51:18.787011] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C652-0-0] [lt=56][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193783700845, cache_obj->added_lc()=false, cache_obj->get_object_id()=13623, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:18.792968] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:18.809278] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=44] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:18.813964] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:18.817201] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=109] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:18.818461] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=62, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:18.818519] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3539, avg_group_log_size=353) [2025-01-08 00:51:18.821056] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1736326278821053, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001085, add_bytes: 0 [2025-01-08 00:51:18.822042] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:18.823495] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2489, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:18.823543] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3539) [2025-01-08 00:51:18.823568] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=233, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:18.823613] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:18.823631] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4015, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:18.823761] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=128] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=91, avg_log_submit_to_flush_time=2794, avg_log_submit_to_slide_time=3924) [2025-01-08 00:51:18.823785] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=34, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:18.823866] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3539, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:18.827336] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38A5-0-0] [lt=325] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:18.827391] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38A5-0-0] [lt=45] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:18.830297] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=613] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:18.830718] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58490][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A924B-0-0] [lt=34] init point thread id with(&point=0x5572d50e47c0, idx_=1146, point=[thread id=58490, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58490) [2025-01-08 00:51:18.831849] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58489][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=21] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-01-08 00:51:18.835382] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4601-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:18.835409] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4601-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31878, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:18.835419] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4601-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=11, avg_log_submit_to_flush_time=2248, avg_log_submit_to_slide_time=31866) [2025-01-08 00:51:18.835452] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:18.835432] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4601-0-0] [lt=11][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18128, log_task={header:{begin_lsn:{lsn:2206435}, end_lsn:{lsn:2206557}, log_id:18128, min_scn:{val:1736326278513691002, v:0}, max_scn:{val:1736326278513691002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2206313}, prev_proposal_id:1, committed_end_lsn:{lsn:2206435}, data_checksum:1715074492, accum_checksum:2171177020, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326278735359, freeze_ts:1736326278735359, submit_ts:1736326278735409, flushed_ts:1736326278743511, gen_to_freeze cost time:0, gen_to_submit cost time:50, submit_to_flush cost time:8102}, fs_cb_begin_ts=1736326278835372, log_life_time=100013) [2025-01-08 00:51:18.835489] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18036, miss_cnt:0, cache_read_size:2206557, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:18.854085] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=35] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:18.859324] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.883983] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=49] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:18.896854] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.896924] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.896953] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.896975] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:18.927540] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38A7-0-0] [lt=77][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18027, log_task={header:{begin_lsn:{lsn:2194158}, end_lsn:{lsn:2194280}, log_id:18027, min_scn:{val:1736326278780077000, v:0}, max_scn:{val:1736326278780077000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2194036}, prev_proposal_id:1, committed_end_lsn:{lsn:2194158}, data_checksum:940009244, accum_checksum:3049872880, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326278827318, freeze_ts:1736326278827318, submit_ts:1736326278827384, flushed_ts:1736326278831716, gen_to_freeze cost time:0, gen_to_submit cost time:66, submit_to_flush cost time:4332}, fs_cb_begin_ts=1736326278927527, log_life_time=100209) [2025-01-08 00:51:18.928839] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1293, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:18.928892] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=51] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:18.928917] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=374, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:18.928986] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=58, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:18.930387] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D80C-0-0] [lt=112] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:18.930475] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D80C-0-0] [lt=86] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31902, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:18.930490] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58465][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D80C-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=26, avg_log_submit_to_flush_time=1718, avg_log_submit_to_slide_time=31876) [2025-01-08 00:51:18.930554] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:18.930657] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17938, miss_cnt:0, cache_read_size:2194402, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:18.951421] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8ACC4-0-0] [lt=206] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.887849187850952148e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=394, Throttle Trigger(MB)=236, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.254635096447166234e+02) [2025-01-08 00:51:18.952087] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=60] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=74, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:18.952370] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:18.953895] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9273) [2025-01-08 00:51:18.955044] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326278750385000, v:0}, new_max_version={val:1736326278750385000, v:0}, last_min_version={val:1736326278401415000, v:0}, last_max_version={val:1736326278401415000, v:0}, total_time_us=5435, delta_us=204656) [2025-01-08 00:51:18.955094] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326278750385000, v:0}, min_version={val:1736326278750385000, v:0}, max_version={val:1736326278750385000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:18.959592] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:18.968726] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread began to shrink(tenant_->id()=1001, group_id_=19, token=0) [2025-01-08 00:51:18.969349] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58213][T1001_L0_G19][T1001][Y0-0000000000000000-0-0] [lt=33] worker thread exit(tenant_->id()=1001, workers_.get_size()=1) [2025-01-08 00:51:18.975161] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=44] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:18.984543] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5241746466, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:18.984580] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:18.984594] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad908599930) [2025-01-08 00:51:18.984603] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5241746466, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:18.989258] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=40] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:19.005158] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=49] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=9273, avg_group_log_size=662) [2025-01-08 00:51:19.005216] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=52] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9273) [2025-01-08 00:51:19.007685] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=2574, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:19.007714] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9273) [2025-01-08 00:51:19.007734] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=305, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:19.007765] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=29, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:19.008727] INFO [LIB] stat (utility.h:1210) [58404][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D00-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=32, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:19.008763] INFO [LIB] stat (utility.h:1210) [58404][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D00-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4149, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:19.008776] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58404][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D00-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=97, avg_log_submit_to_flush_time=2905, avg_log_submit_to_slide_time=4051) [2025-01-08 00:51:19.008948] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:19.008974] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9273, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:19.047747] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9252-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=107, avg_log_submit_to_flush_time=2477, avg_log_submit_to_slide_time=4208) [2025-01-08 00:51:19.047914] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:19.057802] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FD-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:19.057860] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FD-0-0] [lt=44] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326278780077002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326278780077002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:19.059871] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.081316] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=77] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:19.106142] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=32] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:19.110163] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326269104121, end_time=1736326279110161, dag_status={}) [2025-01-08 00:51:19.110264] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=95] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:19.110280] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:19.110288] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:19.110295] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:19.110301] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:19.110307] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:19.110313] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:19.110319] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:19.110338] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:19.110349] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110362] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110373] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110383] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110394] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110413] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110420] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110428] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110435] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110442] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110450] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110457] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110464] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110477] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110504] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110515] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110523] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110531] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110539] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110547] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110554] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110561] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110568] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110581] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110588] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110610] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110621] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110630] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110637] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110645] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110653] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110661] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110669] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110676] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110684] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110692] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110700] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110708] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110716] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110724] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110732] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110740] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110748] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110756] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110764] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110772] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110779] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110787] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110794] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110802] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110810] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110818] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110825] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110832] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110839] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110846] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110853] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110860] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110867] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110874] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110881] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110888] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110894] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110901] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110909] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110916] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110923] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.110930] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.111092] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=160] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.111105] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.111113] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.111121] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:19.111131] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:19.111138] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:19.111145] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:19.111150] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:19.111156] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:19.111161] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:19.111166] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:19.111171] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:19.111271] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=98] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:19.111308] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=37] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:19.111319] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:19.114454] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=36] ====== check clog disk timer task ====== [2025-01-08 00:51:19.114497] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:19.132815] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=503, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:19.152784] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:19.159977] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:19.160026] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.175821] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=98] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6752137216, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:19.186339] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8667, offset=2460041}, end_cursor:ObLogCursor{file_id=1, log_id=8668, offset=2460346}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a60050, len:1655, log_data_len:305, seq:8667, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:19.187307] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E54-0-0] [lt=95] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:19.187357] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E54-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326278631136004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326278631136004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:19.193104] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E54-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326279192231000, start_id=1736326279192231000, end_id=1736326279192231001) [2025-01-08 00:51:19.194425] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:19.196183] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58404][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1001, workers_.get_size()=2) [2025-01-08 00:51:19.200274] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=62] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8668, offset=2460346}, end_cursor:ObLogCursor{file_id=1, log_id=8669, offset=2460643}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a64050, len:1350, log_data_len:297, seq:8668, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:19.200913] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E54-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:19.200971] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E54-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326278631136004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326278631136004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:19.213995] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=98] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8669, offset=2460643}, end_cursor:ObLogCursor{file_id=1, log_id=8670, offset=2460940}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a68050, len:1053, log_data_len:297, seq:8669, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:19.216037] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=37] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8670, offset=2460940}, end_cursor:ObLogCursor{file_id=1, log_id=8671, offset=2461237}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a6c050, len:756, log_data_len:297, seq:8670, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:19.216554] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] no need to update min/max version(new_version={val:1736326278815409000, v:0}, min_version={val:1736326278815409000, v:0}, max_version={val:1736326278815409000, v:0}) [2025-01-08 00:51:19.260158] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.260586] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:19.261527] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A7-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:19.261736] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A7-0-0] [lt=191] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326279015504002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326279015504002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:19.267058] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1}, {id:1001}, {id:1003}]) [2025-01-08 00:51:19.268593] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:19.270289] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:19.272032] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50625][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3BC-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:19.274830] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=87] ====== check clog disk timer task ====== [2025-01-08 00:51:19.274916] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=67] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:19.293204] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=17] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:19.303571] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326278780077003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=523481, in_cluster_service=true, cluster_version={val:1736326278513691004, v:0}, min_cluster_version={val:1736326278513691004, v:0}, max_cluster_version={val:1736326278513691004, v:0}, get_cluster_version_err=0, cluster_version_delta=789867, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18341, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=507221085, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:19.309650] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=98] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:19.312420] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=43] blacklist_loop exec finished(cost_time=72, is_enabled=true, send_cnt=0) [2025-01-08 00:51:19.314260] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-01-08 00:51:19.315626] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=78] gc stale ls task succ [2025-01-08 00:51:19.317007] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:19.317054] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=40] temporary file module use no blocks [2025-01-08 00:51:19.317066] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:19.317080] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:19.317094] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193783230721, flush_io_finished_ret_:0, flush_io_finished_round_:1942, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:19.318988] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=53] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:19.319062] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.319398] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.322772] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:19.323128] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=24] finish to resize log external storage handler(current_ts=1736326279323124, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:19.323155] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=26] snapshot_traffic_info once [2025-01-08 00:51:19.331564] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26] LogLoopThread round_cost_time(us)(round_cost_time=15) [2025-01-08 00:51:19.331604] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:44710, max_lsn:84320539, max_scn:{val:1736326279315890000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84320539}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42377499}}, last_submit_lsn:{lsn:84319953}, last_submit_end_lsn:{lsn:84320539}, last_submit_log_id:44710, last_submit_log_pid:1, max_flushed_lsn:{lsn:84319953}, max_flushed_end_lsn:{lsn:84320539}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84320539}, last_slide_log_id:44710, last_slide_scn:{val:1736326279315890000, v:0}, last_slide_lsn:{lsn:84319953}, last_slide_end_lsn:{lsn:84320539}, last_slide_log_pid:1, last_slide_log_accum_checksum:37980465, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:44711, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84320539}, readable_log_tail:{lsn:84320539}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49888997, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:51:19.331939] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=328] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326279320699, last_flushed_end_lsn:{lsn:84320539}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326279322839, last_flushed_end_lsn:{lsn:84320539}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326279322003, last_flushed_end_lsn:{lsn:84320539}}]) [2025-01-08 00:51:19.331962] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=44711, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326279320699, last_flushed_end_lsn:{lsn:84320539}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326279322839, last_flushed_end_lsn:{lsn:84320539}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326279322003, last_flushed_end_lsn:{lsn:84320539}}]) [2025-01-08 00:51:19.345929] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=67] PNIO [ratelimit] time: 1736326279345925, bytes: 874901792, bw: 0.260002 MB/s, add_ts: 1000066, add_bytes: 272650 [2025-01-08 00:51:19.347663] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=64] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2177740, this={dio_aligned_buf:{buf_write_offset:2886, buf_padding_size:1210, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:22695, truncate_used_ts:5102}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2174976, buf_len=122, total_write_size=11684, total_write_size_after_dio=405504, ob_pwrite_used_ts=687492, count=96, each_pwrite_cost=7161) [2025-01-08 00:51:19.347726] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:19.352953] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:19.354326] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:19.359486] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB5-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:19.359536] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB5-0-0] [lt=45] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:19.360411] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=135][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.361736] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:19.361760] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:19.365284] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870AF-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:19.365329] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:19.365341] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:19.366870] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=55] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:19.366904] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=29] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:19.366916] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:19.366927] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:19.369921] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:19.370930] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=1004] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:19.374020] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=21] current monitor number(seq_=-1) [2025-01-08 00:51:19.376446] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=63] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6752137216, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:19.379443] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] LogLoopThread round_cost_time(us)(round_cost_time=29) [2025-01-08 00:51:19.379979] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=32] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:19.388233] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918542A-0-0] [lt=85][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:19.390890] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC0-0-0] [lt=26] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:19.390930] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC0-0-0] [lt=31] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:19.407283] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58493][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:19.407431] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB61-0-0] [lt=113] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:19.407481] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB61-0-0] [lt=42] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:19.407731] INFO [SERVER.OMT] recv_group_request (ob_tenant.cpp:1413) [46143][pnio1][T0][YB42C0A89F8A-00062B2D1D99BB61-0-0] [lt=18] worker thread created(id()=1001, group->group_id_=19) [2025-01-08 00:51:19.416855] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get_max_applied_scn(scn={val:1736326279315890000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84320539}, palf_committed_end_scn_:{val:1736326279315890000, v:0}, last_check_scn_:{val:1736326279315890000, v:0}, max_applied_cb_scn_:{val:1736326279315890000, v:0}}) [2025-01-08 00:51:19.416900] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84320539}, readable_log_tail:{lsn:84320539}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49888997, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:19.422556] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D056-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:19.435658] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A460D-0-0] [lt=36] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:19.438477] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:51:19.445229] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:19.445279] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:19.445300] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:19.445315] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:19.448585] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:19.450365] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1736326279349095000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43260931}, palf_committed_end_scn_:{val:1736326279403283000, v:0}, last_check_scn_:{val:1736326279403283000, v:0}, max_applied_cb_scn_:{val:1736326279349095000, v:0}}) [2025-01-08 00:51:19.450771] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3170, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:19.450842] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=69] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:19.450862] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=75, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:19.450930] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:19.459530] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.459595] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.460672] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.465141] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:19.465251] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=100] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71543580, tenant_id=1001) [2025-01-08 00:51:19.465278] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:19.471261] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:19.471314] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=51] try gc retain ctx [2025-01-08 00:51:19.471324] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-01-08 00:51:19.471359] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] finish update min start scn(max_decided_scn={val:1736326279315890000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326263731517001, v:0}, keep_alive_scn_:{val:1736326274393594001, v:0}, update_ts_:1736326279470456}) [2025-01-08 00:51:19.472094] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289254996000, latest_log_ts:{val:1736326272617414001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:19.473259] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:19.473284] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] try gc retain ctx [2025-01-08 00:51:19.473291] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] try to retry start_working [2025-01-08 00:51:19.473315] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326279403283000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326263763810001, v:0}, keep_alive_scn_:{val:1736326274427888001, v:0}, update_ts_:1736326279472500}) [2025-01-08 00:51:19.473338] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=42] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:19.473373] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] temporary file module use no blocks [2025-01-08 00:51:19.473384] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:19.473398] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:19.473411] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193783386908, flush_io_finished_ret_:0, flush_io_finished_round_:1938, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:19.474375] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289284252000, latest_log_ts:{val:1736326272625185001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:19.474632] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3539) [2025-01-08 00:51:19.475121] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23] ====== tenant freeze timer task ====== [2025-01-08 00:51:19.481397] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58494][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:19.481461] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=202] worker thread created(tenant_->id()=1001, group_id_=1, token=2) [2025-01-08 00:51:19.499790] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=38] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:19.520023] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=43] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.541685] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.544316] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.545531] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:19.545573] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:19.546005] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:19.547788] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=26] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:19.553142] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:19.553257] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=48] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:19.555955] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC9-0-0] [lt=48] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:19.556219] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC9-0-0] [lt=255] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:19.556495] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58494][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D0A-0-0] [lt=39] init point thread id with(&point=0x5572d50e48c0, idx_=1150, point=[thread id=58494, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58494) [2025-01-08 00:51:19.560987] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.564333] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:51:19.568617] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=37] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:19.576824] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6752137216, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:19.582819] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:19.582915] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44895, [delay/us]:prepare:20, schedule:279, submit:74, rt:1421, total:1795) [2025-01-08 00:51:19.582934] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:19.582991] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44895, [delay/us]: prepare:20, schedule:279, submit:74, device:1421, total:1795) [2025-01-08 00:51:19.583005] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:19.583065] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:15, bw:69383, [delay/us]:prepare:7, schedule:74, submit:301, rt:1585, total:1969) [2025-01-08 00:51:19.583081] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:19.583226] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=79] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:15, avg_bw:69383, [delay/us]: prepare:7, schedule:74, submit:301, device:1585, total:1969) [2025-01-08 00:51:19.583242] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:19.583266] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16325, [delay/us]:prepare:7, schedule:43, submit:1, rt:2342, total:2394) [2025-01-08 00:51:19.583289] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:163254, [delay/us]:prepare:8, schedule:56, submit:503, rt:1732, total:2301) [2025-01-08 00:51:19.583340] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:19.583377] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16325, [delay/us]: prepare:7, schedule:43, submit:1, device:2342, total:2394) [2025-01-08 00:51:19.583393] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:163253, [delay/us]: prepare:8, schedule:56, submit:503, device:1732, total:2301) [2025-01-08 00:51:19.583403] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:19.583424] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4081, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:353, total:364) [2025-01-08 00:51:19.583448] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:19.583475] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4082, [delay/us]: prepare:0, schedule:0, submit:600000000, device:353, total:364) [2025-01-08 00:51:19.583492] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:19.583508] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:19.583516] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:19.583524] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:19.583533] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=261kB/s, net_out=219kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:19.626538] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8B-0-0] [lt=10] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326279728532, keepalive_ts:-1}) [2025-01-08 00:51:19.626611] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8B-0-0] [lt=70] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326279728532, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:19.632938] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:19.633164] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=58] get number(ret=0, service_type_=0, range=1, base_id=1736326279627356000, start_id=1736326279627356000, end_id=1736326279627356001) [2025-01-08 00:51:19.633289] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58306][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=7] worker thread exit(tenant_->id()=1002, workers_.get_size()=6) [2025-01-08 00:51:19.650475] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:19.650528] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=51] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4952, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:19.661275] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.671545] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=166] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3653) [2025-01-08 00:51:19.671578] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:19.676321] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C9E-0-0] [lt=54][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:19.678520] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C9E-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:19.683907] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=34] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:19.683954] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=46] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:19.684011] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84985-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:19.684031] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84985-0-0] [lt=17] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:19.686213] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=29] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:19.737384] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:19.738487] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=34, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:19.738508] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:19.739960] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=120] get number(ret=0, service_type_=0, range=1, base_id=1736326279739822000, start_id=1736326279739822000, end_id=1736326279739822001) [2025-01-08 00:51:19.741450] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=91] success to update min reserved snapshot(reserved_snapshot=1736324475608151000, duration=1800, snapshot_gc_ts=1736326275608151000) [2025-01-08 00:51:19.746837] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38B7-0-0] [lt=45] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:19.746881] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38B7-0-0] [lt=29][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18035, log_task={header:{begin_lsn:{lsn:2195125}, end_lsn:{lsn:2195247}, log_id:18035, min_scn:{val:1736326279275750003, v:0}, max_scn:{val:1736326279275750003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2195003}, prev_proposal_id:1, committed_end_lsn:{lsn:2195125}, data_checksum:940009244, accum_checksum:1332431646, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326279646792, freeze_ts:1736326279646792, submit_ts:1736326279646798, flushed_ts:1736326279650333, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:3535}, fs_cb_begin_ts=1736326279746872, log_life_time=100080) [2025-01-08 00:51:19.747424] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/71:182382 disk:308K/71:182382 t1m2:48K/11:781297 t1m2g0:48K/11:781297 t1002m2:188K/45:65536 t1002m2g0:188K/45:65536 t1001m2:72K/15:93715 t1001m2g0:72K/15:93715 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:19.747487] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18686039, this={dio_aligned_buf:{buf_write_offset:632, buf_padding_size:3464, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:32483, truncate_used_ts:6714}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=18685952, buf_len=545, total_write_size=16019, total_write_size_after_dio=430080, ob_pwrite_used_ts=318080, count=101, each_pwrite_cost=3149) [2025-01-08 00:51:19.747601] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=67, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:19.747681] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=70] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1755, avg_group_log_size=159) [2025-01-08 00:51:19.750879] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1755) [2025-01-08 00:51:19.751269] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326279552683000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=198541, in_cluster_service=true, cluster_version={val:1736326279349095000, v:0}, min_cluster_version={val:1736326279349095000, v:0}, max_cluster_version={val:1736326279349095000, v:0}, get_cluster_version_err=0, cluster_version_delta=402129, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18780, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=625388007, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:19.753516] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:19.754066] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:19.758108] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58496][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:19.758722] INFO [LIB] stat (utility.h:1210) [58490][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9269-0-0] [lt=49] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:19.760049] INFO [LIB] stat (utility.h:1210) [58490][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9269-0-0] [lt=1324] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6150, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:19.761572] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.761984] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58496][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9268-0-0] [lt=37] init point thread id with(&point=0x5572d50e4940, idx_=1152, point=[thread id=58496, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58496) [2025-01-08 00:51:19.763329] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread created(tenant_->id()=1002, group_id_=1, token=6) [2025-01-08 00:51:19.771304] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46384][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C654-0-0] [lt=101][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193784685136, cache_obj->added_lc()=false, cache_obj->get_object_id()=13624, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:19.772010] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=38] update reserved snapshot finished(cost_ts=30563, reserved_snapshot=1736324475608151000) [2025-01-08 00:51:19.775450] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58497][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:19.778046] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=16] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:19.778477] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97] worker thread created(tenant_->id()=1001, group_id_=1, token=3) [2025-01-08 00:51:19.778842] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE872-0-0] [lt=81] get number(ret=0, service_type_=0, range=1, base_id=1736326279778406000, start_id=1736326279778406000, end_id=1736326279778406001) [2025-01-08 00:51:19.790573] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=47] ====== check clog disk timer task ====== [2025-01-08 00:51:19.790614] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=36] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:19.793461] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:19.806163] INFO pn_ratelimit (group.c:653) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=43] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:19.806275] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AD5A-0-0] [lt=108] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=109) [2025-01-08 00:51:19.809261] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=643518464, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=430223360, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:19.809403] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=46] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:19.809427] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=22] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2841, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:19.809491] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:19.809544] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:19.809729] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=183] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:19.810324] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=246] [APPLY STAT CB APPEND COST TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=132, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:19.810352] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=28] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=4728, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:19.810362] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:19.810369] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=-788, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:19.810377] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=4072, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:19.810396] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:19.810530] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=36] ====== checkpoint timer task ====== [2025-01-08 00:51:19.811840] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=134] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:19.811954] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=112] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:19.812061] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=27] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289284252000, latest_log_ts:{val:1736326272625185001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:19.812177] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=111] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:19.812219] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=35] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:19.812349] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=125] get rec log scn(service_type_=0, rec_log_ts={val:1736326272625185001, v:0}) [2025-01-08 00:51:19.812359] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:19.812367] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:19.812378] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:19.812396] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:19.812433] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=33] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:19.812457] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:19.812479] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:19.812489] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:19.814484] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=37] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:19.818155] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=89] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:19.819143] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:19.820459] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=100] get number(ret=0, service_type_=0, range=1, base_id=1736326279817405000, start_id=1736326279817405000, end_id=1736326279817405001) [2025-01-08 00:51:19.820872] INFO [LIB] stat (utility.h:1210) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=101, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:19.821029] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3490, avg_group_log_size=349) [2025-01-08 00:51:19.823653] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:19.824106] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA688D-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:19.824207] INFO [LIB] stat (utility.h:1210) [58461][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA688D-0-0] [lt=99] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4992, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:19.826692] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1736326279826688, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005635, add_bytes: 0 [2025-01-08 00:51:19.833439] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=290] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:19.833650] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58436][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=60] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-01-08 00:51:19.835061] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4615-0-0] [lt=104] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:19.835097] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4615-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:19.836644] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=58] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1716, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:19.836671] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:19.836690] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=197, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:19.836811] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:19.842730] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38B9-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:19.842771] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38B9-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:19.844770] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=29] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:19.846953] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=33] get number(ret=0, service_type_=0, range=1, base_id=1736326279846662000, start_id=1736326279846662000, end_id=1736326279846662001) [2025-01-08 00:51:19.847846] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [58495][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=0] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5242884188, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:51:19.849696] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=2617, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:19.849724] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=190, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:19.849746] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=33, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:19.849995] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [58495][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=77] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5242884188, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:51:19.854927] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=73] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:19.861713] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=123][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.892376] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:51:19.896768] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.896845] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.896893] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.896938] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:19.901385] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=27] ====== checkpoint timer task ====== [2025-01-08 00:51:19.902703] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=51] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:19.902802] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=97] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:19.903109] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=38] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289254996000, latest_log_ts:{val:1736326272617414001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:19.903234] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=119] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289254996000, latest_log_ts:{val:1736326272617414001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:19.903441] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=198] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:19.903597] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=152] get rec log scn(service_type_=0, rec_log_ts={val:1736326272617414001, v:0}) [2025-01-08 00:51:19.903612] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=15] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:19.903619] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:19.903642] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:19.903732] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=81] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:19.903753] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:19.903774] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:19.903862] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=86] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:19.903877] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:19.935547] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4617-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:19.935626] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4617-0-0] [lt=78] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20834, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:19.935653] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4617-0-0] [lt=25] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=1941, avg_log_submit_to_slide_time=20814) [2025-01-08 00:51:19.935670] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4617-0-0] [lt=16][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18139, log_task={header:{begin_lsn:{lsn:2207777}, end_lsn:{lsn:2207899}, log_id:18139, min_scn:{val:1736326279516066003, v:0}, max_scn:{val:1736326279516066003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2207655}, prev_proposal_id:1, committed_end_lsn:{lsn:2207777}, data_checksum:1715074492, accum_checksum:3467011472, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326279835050, freeze_ts:1736326279835050, submit_ts:1736326279835089, flushed_ts:1736326279836759, gen_to_freeze cost time:0, gen_to_submit cost time:39, submit_to_flush cost time:1670}, fs_cb_begin_ts=1736326279935534, log_life_time=100484) [2025-01-08 00:51:19.935958] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:19.936035] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18047, miss_cnt:0, cache_read_size:2207899, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:19.936149] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] succ to submit log task to replay service(to_submit_lsn={lsn:2207833}, to_submit_scn={val:1736326279516066003, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326279935539, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2207899}, next_to_submit_scn_:{val:1736326279516066004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2207777}, end_lsn:{lsn:2207899}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2207899}, readable_log_tail:{lsn:2207899}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64896869, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326279516066003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326279516066003, v:0}, data_checksum:3348837161, flag:6111}}, init_mode_version:0, accumulate_checksum:3467011472, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326279516066003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18047, miss_cnt:0, cache_read_size:2207899, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:19.944305] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38BB-0-0] [lt=31] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:19.944420] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38BB-0-0] [lt=112] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=25939, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:19.944439] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38BB-0-0] [lt=17] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=2269, avg_log_submit_to_slide_time=25925) [2025-01-08 00:51:19.944507] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38BB-0-0] [lt=67][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18037, log_task={header:{begin_lsn:{lsn:2195369}, end_lsn:{lsn:2195491}, log_id:18037, min_scn:{val:1736326279776878000, v:0}, max_scn:{val:1736326279776878000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2195247}, prev_proposal_id:1, committed_end_lsn:{lsn:2195369}, data_checksum:940009244, accum_checksum:1741043388, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326279842722, freeze_ts:1736326279842722, submit_ts:1736326279842763, flushed_ts:1736326279848123, gen_to_freeze cost time:0, gen_to_submit cost time:41, submit_to_flush cost time:5360}, fs_cb_begin_ts=1736326279944295, log_life_time=101573) [2025-01-08 00:51:19.945261] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=101] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1089, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:19.945322] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17947, miss_cnt:0, cache_read_size:2195491, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:19.954437] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8ACD3-0-0] [lt=45] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.888000535964965820e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=394, Throttle Trigger(MB)=236, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.254081814799674248e+02) [2025-01-08 00:51:19.954926] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=74] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=62, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:19.954965] INFO [ARCHIVE] load_archive_round_attr (ob_archive_persist_mgr.cpp:323) [50466][T1002_ArcSrv][T1002][YB42C0A89F89-00062B2D1C48458E-0-0] [lt=22] no archive dest exist, just skip(ret=-4018, tenant_id_=1002) [2025-01-08 00:51:19.956511] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=24] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12807819264, fast_delete_log_mode=false) [2025-01-08 00:51:19.956725] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=3, inc_cnt=1, queue_size=3) [2025-01-08 00:51:19.957838] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:19.959778] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [58499][IO_SYNC_CH3][T0][Y0-0000000000000000-0-0] [lt=0] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:4, threads_idle_time:5243135963, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:19.960388] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [58499][IO_SYNC_CH3][T0][Y0-0000000000000000-0-0] [lt=47] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:5, threads_idle_time:5243135963, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:19.964603] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=125] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:19.965069] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:19.978909] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=87] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:20.008146] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9931) [2025-01-08 00:51:20.065271] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.134097] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=84] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:20.165386] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.260924] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:20.265614] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.269482] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=47] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:20.317309] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=71] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:20.317377] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=59] temporary file module use no blocks [2025-01-08 00:51:20.317392] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:20.317410] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:20.317505] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=93] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193784231010, flush_io_finished_ret_:0, flush_io_finished_round_:1943, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:20.319457] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.349625] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=47] PNIO [ratelimit] time: 1736326280349621, bytes: 875023153, bw: 0.115313 MB/s, add_ts: 1003696, add_bytes: 121361 [2025-01-08 00:51:20.356634] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F8B-00062B2D18EC3348-0-0] [lt=15][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=310779, send_timestamp=1736326280045852, peer="192.168.159.139:45074", sz=265) [2025-01-08 00:51:20.362200] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB6-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:20.362349] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB6-0-0] [lt=137] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:20.365819] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.374350] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=90] current monitor number(seq_=-1) [2025-01-08 00:51:20.417114] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:20.408, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.438626] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=44] gc stale ls task succ [2025-01-08 00:51:20.440161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:20.433, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.458313] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.459582] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.460864] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=143][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:20.452, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.466484] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.501837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:20.497, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.512115] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:20.502, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.547027] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.547461] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.566714] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.585691] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:20.585839] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12251, [delay/us]:prepare:81, schedule:2010, submit:50, rt:1381, total:3523) [2025-01-08 00:51:20.585869] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:20.585945] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=51] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:3, avg_bw:12251, [delay/us]: prepare:81, schedule:2010, submit:50, device:1381, total:3523) [2025-01-08 00:51:20.585963] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:20.586021] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20419, [delay/us]:prepare:7, schedule:570, submit:916, rt:3816, total:5310) [2025-01-08 00:51:20.586037] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=1, iops=1, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:20.586096] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20420, [delay/us]: prepare:7, schedule:570, submit:916, device:3816, total:5310) [2025-01-08 00:51:20.586112] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:20.586165] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:4, hold_mem:32768 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4973, iops:14, bw:69432, [delay/us]:prepare:12, schedule:215, submit:331, rt:2174, total:2734) [2025-01-08 00:51:20.586181] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1002, ref_cnt=10, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49996, free_result_cnt=49996, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:20.586220] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4974, avg_iops:14, avg_bw:69435, [delay/us]: prepare:12, schedule:215, submit:331, device:2174, total:2734) [2025-01-08 00:51:20.586231] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:20.586371] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=125] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4083, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:944, total:958) [2025-01-08 00:51:20.586403] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:20.586451] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4083, [delay/us]: prepare:0, schedule:0, submit:600000000, device:944, total:958) [2025-01-08 00:51:20.586475] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:20.586496] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=1, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 1 [2025-01-08 00:51:20.586504] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:20.586524] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:20.586536] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=80kB/s, net_out=89kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:20.609819] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe410, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfe6f0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:20.591, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.649054] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=105][errcode=-4016] task executed too delay(delay_time=487042, *p_task_={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:0, time_interval:250.00ms, expected_run_time:00:51:20.658, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.649142] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=72][errcode=-4016] task executed too delay(delay_time=252414, *p_task_={this:0x2ad8a0dff190, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff470, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:0, time_interval:500.00ms, expected_run_time:00:51:20.882, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.649183] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=31] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:20.649385] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8B-00062B2D18EC3349-0-0] [lt=20][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=503096, send_timestamp=1736326280146285, peer="192.168.159.139:45082", sz=265) [2025-01-08 00:51:20.649915] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=-4016] task executed too delay(delay_time=547340, *p_task_={this:0x2ad8f0dfe410, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfe6f0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:51:21.919, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.650056] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4016] task executed too delay(delay_time=455318, *p_task_={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:20.683, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.650156] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4016] task executed too delay(delay_time=434960, *p_task_={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:20.702, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.650278] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=111][errcode=-4016] task executed too delay(delay_time=394261, *p_task_={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:20.747, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.650424] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=108][errcode=-4016] task executed too delay(delay_time=384247, *p_task_={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:20.752, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.650529] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] task executed too delay(delay_time=364058, *p_task_={this:0x2ad8effff190, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff470, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:51:20.770, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.650615] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] task executed too delay(delay_time=159019, *p_task_={this:0x2ad8f0dff570, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:51:20.989, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.650649] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4016] task executed too delay(delay_time=138545, *p_task_={this:0x2ad8f0dfc950, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfcc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:51:21.510, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.650877] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F8B-00062B2D1D7BF1B1-0-0] [lt=70][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=602663, send_timestamp=1736326280048212, peer="192.168.159.139:45062", sz=265) [2025-01-08 00:51:20.652133] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46308][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=23] expand thread count(this=0x2ad8881fc3b0, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:20.652946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=53][errcode=0] task scheduled too delay(scheduled_delay=627920, expected_run_ts_us_=1736326280025022, ObClockGenerator::getRealClock()=1736326280652943, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:20.250, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.653068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=91][errcode=0] task scheduled too delay(scheduled_delay=386717, expected_run_ts_us_=1736326280266349, ObClockGenerator::getRealClock()=1736326280653067, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950ef730, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efa10, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:500.00ms, expected_run_time:00:51:20.266, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.653204] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46141][pnio1][T0][YB42C0A89F8B-00062B2D1BBBF321-0-0] [lt=41][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=605910, send_timestamp=1736326280047292, peer="192.168.159.139:45058", sz=265) [2025-01-08 00:51:20.653816] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE875-0-0] [lt=106] get number(ret=0, service_type_=0, range=1, base_id=1736326280647139000, start_id=1736326280647139000, end_id=1736326280647139001) [2025-01-08 00:51:20.653987] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDB-0-0] [lt=31] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:20.654011] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDB-0-0] [lt=17] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=29) [2025-01-08 00:51:20.654584] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [50419][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=62] expand thread count(this=0x2ad8c0eb0e70, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:20.655156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50522][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] task scheduled too delay(scheduled_delay=709847, expected_run_ts_us_=1736326279945306, ObClockGenerator::getRealClock()=1736326280655155, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfcdb0, caller:mds_tenant_service.cpp:mtl_start:125, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8f0dfd090, time_wheel:0x2ad8dacdf990, thread_pool:0x2ad8dacdf170, is_running:1, time_interval:3.00s, expected_run_time:00:51:19.945, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.655371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] task scheduled too delay(scheduled_delay=606013, expected_run_ts_us_=1736326280049355, ObClockGenerator::getRealClock()=1736326280655369, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:20.493, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:20.655472] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=95] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:20.658036] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=7] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:20.658066] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=19] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:20.658081] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=97) [2025-01-08 00:51:20.658366] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BA1-0-0] [lt=44] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:20.663348] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:20.664534] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:20.664600] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=57] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:20.664615] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=11] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:20.664638] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=21] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:20.665745] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=817882, clean_num=31457) [2025-01-08 00:51:20.665784] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:20.666252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.666452] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=81] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.666906] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.668298] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4015] clock out of order(cur_ts=1736326280668295, cur_ts_=1736326280647139, delta=21156) [2025-01-08 00:51:20.668332] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1}, {id:1002}]) [2025-01-08 00:51:20.668403] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:20.668430] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:20.668488] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:51:20.669013] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=70258, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:20.669085] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=69][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84322491}], scn_array=[cnt:1, {val:1736326279919756000, v:0}], curr_size=586, accum_size=4076, time_cost=748046) [2025-01-08 00:51:20.669115] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4076) [2025-01-08 00:51:20.669139] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=354, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:20.669178] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1952) [2025-01-08 00:51:20.669367] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=102] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=33, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:20.671776] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:20.671818] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:20.673042] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870B2-0-0] [lt=11][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:20.673096] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:20.673109] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:20.674467] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BF6-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=128, avg_log_submit_to_flush_time=70682, avg_log_submit_to_slide_time=72930) [2025-01-08 00:51:20.674495] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58461][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5BF6-0-0] [lt=27][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=44716, log_task={header:{begin_lsn:{lsn:84322491}, end_lsn:{lsn:84323077}, log_id:44716, min_scn:{val:1736326279919756000, v:0}, max_scn:{val:1736326279919756000, v:0}, data_len:530, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84322491}, data_checksum:2380445720, accum_checksum:3133475540, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326279920731, freeze_ts:1736326279920748, submit_ts:1736326279920828, flushed_ts:1736326280669165, gen_to_freeze cost time:17, gen_to_submit cost time:97, submit_to_flush cost time:748337}, fs_cb_begin_ts=1736326280674453, log_life_time=753722) [2025-01-08 00:51:20.674619] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918542B-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:20.674809] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:20.674833] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:51:20.674843] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:20.674856] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:20.674869] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193784387284, flush_io_finished_ret_:0, flush_io_finished_round_:1939, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:20.674877] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=32][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326279919756000, v:0}, log_ts:{val:1736326279919756000, v:0}, lsn:{lsn:84322547}, submit_ts:1736326279920674}, this:0x2ad89c0a3bd8, is_inited_:true, trans_id:{txid:27985}, ls_id:{id:1}, ctx:0x2ad89c0a01d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=754198, ctx_lock_wait_time=1) [2025-01-08 00:51:20.674909] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=41] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:20.674924] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-01-08 00:51:20.674930] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:20.674939] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:20.674948] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193784397123, flush_io_finished_ret_:0, flush_io_finished_round_:3767, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:20.674976] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=42] [APPLY STAT CB APPEND COST TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=138, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:20.674989] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=13] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=21526, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:20.674997] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:20.675004] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=6] [APPLY STAT CB EXECUTE TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=-1536, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:20.675010] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=20129, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:20.675037] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=111] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4076, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:20.675169] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=79] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326279716877000, v:0}, new_max_version={val:1736326279716877000, v:0}, last_min_version={val:1736326279315890000, v:0}, last_max_version={val:1736326279315890000, v:0}, total_time_us=757380, delta_us=958290) [2025-01-08 00:51:20.675192] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326279716877000, v:0}, min_version={val:1736326279716877000, v:0}, max_version={val:1736326279716877000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:20.675251] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:20.676766] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D2248464A-0-0] [lt=78] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:51:20.677244] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18682722}], scn_array=[cnt:1, {val:1736326279739822003, v:0}], curr_size=122, accum_size=244, time_cost=729636) [2025-01-08 00:51:20.677304] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=75133, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:20.677320] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2195491}], scn_array=[cnt:1, {val:1736326279776878001, v:0}], curr_size=122, accum_size=1211, time_cost=732090) [2025-01-08 00:51:20.677334] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:20.677345] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=192, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:20.677367] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2207899}], scn_array=[cnt:1, {val:1736326279516066004, v:0}], curr_size=122, accum_size=122, time_cost=741534) [2025-01-08 00:51:20.677398] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=50692, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:20.677408] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43263945}], scn_array=[cnt:1, {val:1736326279951011000, v:0}], curr_size=589, accum_size=9342, time_cost=721622) [2025-01-08 00:51:20.677420] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9342) [2025-01-08 00:51:20.677434] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=7] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=295, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:20.677454] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=124171, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:20.677466] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2174488}], scn_array=[cnt:1, {val:1736326279739822003, v:0}], curr_size=122, accum_size=722, time_cost=729734) [2025-01-08 00:51:20.677479] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=722) [2025-01-08 00:51:20.677488] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=237, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:20.678871] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=60] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9931) [2025-01-08 00:51:20.678945] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9997, avg_group_log_size=624) [2025-01-08 00:51:20.679005] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=54] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=27, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:20.680743] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BA3-0-0] [lt=37] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:20.680889] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BA5-0-0] [lt=23] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:20.683446] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9270-0-0] [lt=33] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1145) [2025-01-08 00:51:20.686647] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C9F-0-0] [lt=52][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:20.688084] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C9F-0-0] [lt=42][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:20.688664] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=623, avg_log_gen_to_submit_time=708, avg_log_submit_to_flush_time=77040, avg_log_submit_to_slide_time=79646) [2025-01-08 00:51:20.688691] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=19501, log_task={header:{begin_lsn:{lsn:18682722}, end_lsn:{lsn:18682844}, log_id:19501, min_scn:{val:1736326279739822003, v:0}, max_scn:{val:1736326279739822003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18682722}, data_checksum:667309872, accum_checksum:2482056066, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326279947404, freeze_ts:1736326279947410, submit_ts:1736326279947490, flushed_ts:1736326280688633, gen_to_freeze cost time:6, gen_to_submit cost time:86, submit_to_flush cost time:741143}, fs_cb_begin_ts=1736326280688649, log_life_time=741245) [2025-01-08 00:51:20.688786] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=59, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:20.688846] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=844) [2025-01-08 00:51:20.688865] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:20.688897] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1633, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:20.689110] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:20.689552] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:20.689700] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:20.689724] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=20] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:20.689735] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=9] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:20.689744] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=7] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:20.692270] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [46340][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=40] XA scheduler heartbeat task statistics(avg_time=0) [2025-01-08 00:51:20.692815] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=22] [APPLY STAT CB APPEND COST TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=139, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:20.692840] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=24] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=31182, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:20.692965] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=123] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:20.692975] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB EXECUTE TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=-1257, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:20.692982] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=30064, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:20.693058] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:20.693160] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:51:20.694337] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=1172] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:20.694468] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] success to update min reserved snapshot(reserved_snapshot=1736324474317657000, duration=1800, snapshot_gc_ts=1736326274317657000) [2025-01-08 00:51:20.694536] INFO [LIB] stat (utility.h:1210) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:20.694614] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=34] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:20.695425] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=100] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:5, threads_idle_time:15042901926, tenant_id:1}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:20.701289] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F510-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326280694597000, start_id=1736326280694597000, end_id=1736326280694597001) [2025-01-08 00:51:20.707621] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EE7-0-0] [lt=64] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:20.712478] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A926E-0-0] [lt=220] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:20.712505] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A926E-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=195145, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:20.712538] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A926E-0-0] [lt=31][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=17866, log_task={header:{begin_lsn:{lsn:2174488}, end_lsn:{lsn:2174610}, log_id:17866, min_scn:{val:1736326279739822003, v:0}, max_scn:{val:1736326279739822003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2174488}, data_checksum:2222604892, accum_checksum:612568084, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326279947514, freeze_ts:1736326279947518, submit_ts:1736326279947572, flushed_ts:1736326280688839, gen_to_freeze cost time:4, gen_to_submit cost time:58, submit_to_flush cost time:741267}, fs_cb_begin_ts=1736326280712458, log_life_time=764944) [2025-01-08 00:51:20.713198] INFO [LIB] stat (utility.h:1210) [58474][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39A7-0-0] [lt=23] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=22, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:20.713482] INFO [LIB] stat (utility.h:1210) [58474][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39A7-0-0] [lt=281] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=55297, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:20.713503] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58474][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39A7-0-0] [lt=18] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=143, avg_log_gen_to_submit_time=227, avg_log_submit_to_flush_time=51118, avg_log_submit_to_slide_time=55069) [2025-01-08 00:51:20.713547] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58474][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39A7-0-0] [lt=42][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=32708, log_task={header:{begin_lsn:{lsn:43263945}, end_lsn:{lsn:43264534}, log_id:32708, min_scn:{val:1736326279951011000, v:0}, max_scn:{val:1736326279951011000, v:0}, data_len:533, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43263945}, data_checksum:2857235825, accum_checksum:2958038544, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326279954857, freeze_ts:1736326279954861, submit_ts:1736326279955013, flushed_ts:1736326280678863, gen_to_freeze cost time:4, gen_to_submit cost time:156, submit_to_flush cost time:723850}, fs_cb_begin_ts=1736326280713163, log_life_time=758306) [2025-01-08 00:51:20.714027] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9342, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:20.714262] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50623][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3BF-0-0] [lt=20][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:20.714776] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326279951011000, v:0}, log_ts:{val:1736326279951011000, v:0}, lsn:{lsn:43264001}, submit_ts:1736326279954850}, this:0x2ad8fb7086d8, is_inited_:true, trans_id:{txid:66842}, ls_id:{id:1}, ctx:0x2ad8fb704cd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=759920, ctx_lock_wait_time=2) [2025-01-08 00:51:20.714919] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=74][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326280007355000, v:0}, log_ts:{val:1736326280007355000, v:0}, lsn:{lsn:43264590}, submit_ts:1736326280008126}, this:0x2ad8fb6c0b58, is_inited_:true, trans_id:{txid:66843}, ls_id:{id:1}, ctx:0x2ad8fb6bd150, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=706789, ctx_lock_wait_time=0) [2025-01-08 00:51:20.715638] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=146] get_max_applied_scn(scn={val:1736326279739822002, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2174610}, palf_committed_end_scn_:{val:1736326279739822003, v:0}, last_check_scn_:{val:1736326279739822003, v:0}, max_applied_cb_scn_:{val:1736326279739822002, v:0}}) [2025-01-08 00:51:20.715679] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=38] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2174610}, readable_log_tail:{lsn:2174610}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64930158, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:20.725836] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=24] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:20.730674] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43265189}], scn_array=[cnt:1, {val:1736326280133031001, v:0}], curr_size=112, accum_size=767, time_cost=11423) [2025-01-08 00:51:20.730833] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D059-0-0] [lt=34][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:20.734508] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [50419][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=28] expand thread count(this=0x2ad8c0eb0e70, max_thread_cnt=128, cur_thread_count=5, inc_cnt=1, queue_size=5) [2025-01-08 00:51:20.736793] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=31] update reserved snapshot finished(cost_ts=42329, reserved_snapshot=1736324474317657000) [2025-01-08 00:51:20.738643] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EE8-0-0] [lt=47] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:20.738954] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58505][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:20.739295] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [50419][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=65] expand thread count(this=0x2ad8c0eb0e70, max_thread_cnt=128, cur_thread_count=6, inc_cnt=1, queue_size=6) [2025-01-08 00:51:20.740335] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=36] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:20.741933] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=94] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:20.741962] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=28] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:20.744237] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487A-0-0] [lt=9] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:20.744303] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487A-0-0] [lt=18] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:20.744318] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487A-0-0] [lt=11] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:20.745070] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:20.746804] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:20.746897] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326274317657000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:20.746924] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326274317657000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326274317657000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:20.746999] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:7, threads_idle_time:7834851297, tenant_id:1002}, cur_thread_count=7, cnt=1, new_thread_count=8) [2025-01-08 00:51:20.747288] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=14] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:20.749165] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=19] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:20.752677] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EE9-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:20.754727] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:20.754754] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:20.754769] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:20.754777] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:20.754784] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:20.754792] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=96) [2025-01-08 00:51:20.760575] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=30] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:3, threads_idle_time:15042918671, tenant_id:1}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:20.760632] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=12478464) [2025-01-08 00:51:20.760804] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:20.762395] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=77] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:20.762491] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326275608151000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:20.762524] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:51:20.762535] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=1) [2025-01-08 00:51:20.762544] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:51:20.762553] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:51:20.764314] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=63, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:20.764343] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=3, avg_log_batch_cnt=1, total_group_log_size=366, avg_group_log_size=122) [2025-01-08 00:51:20.764360] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1145) [2025-01-08 00:51:20.764374] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:20.764419] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=32, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:20.764437] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=7, avg_log_batch_cnt=1, total_group_log_size=844, avg_group_log_size=120) [2025-01-08 00:51:20.764448] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=844) [2025-01-08 00:51:20.764458] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:20.764632] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326276666280, last_first_register_time_us=1736324449183670, register_time_us=1736326276666280, curr_time_us=1736326280764629) [2025-01-08 00:51:20.764694] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=34] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326280764668, last_submit_register_req_time_us=1736326280764668, register_parent_reason=1) [2025-01-08 00:51:20.764713] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:20.764769] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:20.766340] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58490][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF54-0-0] [lt=40] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326280857301, keepalive_ts:-1}) [2025-01-08 00:51:20.766362] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58490][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF54-0-0] [lt=21] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326280857301, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:20.766440] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58490][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4619-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=732) [2025-01-08 00:51:20.766467] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58490][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4619-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18140, log_task={header:{begin_lsn:{lsn:2207899}, end_lsn:{lsn:2208021}, log_id:18140, min_scn:{val:1736326279516066004, v:0}, max_scn:{val:1736326279516066004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2207777}, prev_proposal_id:1, committed_end_lsn:{lsn:2207899}, data_checksum:1715074492, accum_checksum:2369068988, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326279935510, freeze_ts:1736326279935510, submit_ts:1736326279935517, flushed_ts:1736326280688806, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:753289}, fs_cb_begin_ts=1736326280766457, log_life_time=830947) [2025-01-08 00:51:20.766523] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58490][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B9-0-0] [lt=27] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326280757023, keepalive_ts:-1}) [2025-01-08 00:51:20.766536] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58490][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B9-0-0] [lt=13] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326280757023, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:20.766659] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=213, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:20.766679] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:20.767787] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.767972] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58490][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B8-0-0] [lt=9] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326280756852, keepalive_ts:-1}) [2025-01-08 00:51:20.768039] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58490][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2B8-0-0] [lt=65] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326280756852, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:20.768863] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58468][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F5-0-0] [lt=35] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326280759371, keepalive_ts:-1}) [2025-01-08 00:51:20.768905] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58468][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F5-0-0] [lt=40] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326280759371, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:20.768962] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=38] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:84K/20:473072 disk:84K/20:473072 t1m2:12K/3:17801 t1m2g0:12K/3:17801 t1002m2:48K/11:598858 t1002m2g0:48K/11:598858 t1001m2:24K/6:468124 t1001m2g0:24K/6:468124 |q0: 0 1 0 |q1: 0 0 0 [2025-01-08 00:51:20.770911] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=66] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:20.771454] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58497][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2C-0-0] [lt=39] init point thread id with(&point=0x5572d50e4980, idx_=1153, point=[thread id=58497, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58497) [2025-01-08 00:51:20.771531] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58497][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2C-0-0] [lt=69] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326280755806, keepalive_ts:-1}) [2025-01-08 00:51:20.771609] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58497][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2C-0-0] [lt=76] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326280755806, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:20.772961] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EEA-0-0] [lt=27] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:20.773631] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8636-0-0] [lt=76] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=366) [2025-01-08 00:51:20.773884] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8636-0-0] [lt=20][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18038, log_task={header:{begin_lsn:{lsn:2195491}, end_lsn:{lsn:2195613}, log_id:18038, min_scn:{val:1736326279776878001, v:0}, max_scn:{val:1736326279776878001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2195369}, prev_proposal_id:1, committed_end_lsn:{lsn:2195491}, data_checksum:940009244, accum_checksum:2679155557, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326279944265, freeze_ts:1736326279944265, submit_ts:1736326279944272, flushed_ts:1736326280688737, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:744465}, fs_cb_begin_ts=1736326280773657, log_life_time=829392) [2025-01-08 00:51:20.774329] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1961) [2025-01-08 00:51:20.774357] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:20.774584] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=23] failed to get result(ret=-4008) [2025-01-08 00:51:20.774697] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=16] check new jobs(ret=0, tenant_id=1, is_oracle_tenant=false, job_infos=[{tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326178662155, start_date_:1736323478339648, last_date_:1736326178341674, this_date_:0, next_date_:1736327078339648, end_date_:64060560000000000, total_:8578451, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736408978339542, last_date_:0, this_date_:0, next_date_:1736408978339542, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:51:20.774820] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=121] check all tenants(ret=0) [2025-01-08 00:51:20.776517] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8671, offset=2461237}, end_cursor:ObLogCursor{file_id=1, log_id=8672, offset=2461542}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a70050, len:459, log_data_len:305, seq:8671, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:20.776958] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18682844}], scn_array=[cnt:1, {val:1736326280628634000, v:0}], curr_size=122, accum_size=366, time_cost=12051) [2025-01-08 00:51:20.777001] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=366) [2025-01-08 00:51:20.777308] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:20.777350] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=253340, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:20.777548] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2174610}], scn_array=[cnt:1, {val:1736326280628634000, v:0}], curr_size=122, accum_size=122, time_cost=12644) [2025-01-08 00:51:20.778655] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=7740] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:20.781147] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487A-0-0] [lt=119] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:20.781394] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487A-0-0] [lt=181] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:20.781597] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487A-0-0] [lt=198] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:20.785665] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D3F-0-0] [lt=15] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:20.786709] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EEC-0-0] [lt=7] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:20.787591] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5243631776, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:20.787616] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:20.787627] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a0420) [2025-01-08 00:51:20.787635] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5243631776, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:20.787657] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=24] ====== tenant freeze timer task ====== [2025-01-08 00:51:20.788010] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58505][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39AA-0-0] [lt=23] init point thread id with(&point=0x5572d50e4b80, idx_=1161, point=[thread id=58505, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58505) [2025-01-08 00:51:20.788341] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=41] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:20.788527] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E55-0-0] [lt=46] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:20.788586] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E55-0-0] [lt=39] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326279739822003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326279739822003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:20.789415] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=67][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43265301}], scn_array=[cnt:1, {val:1736326280628634000, v:0}], curr_size=122, accum_size=889, time_cost=10684) [2025-01-08 00:51:20.789572] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84323199}], scn_array=[cnt:1, {val:1736326280628634000, v:0}], curr_size=122, accum_size=244, time_cost=15166) [2025-01-08 00:51:20.791060] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EED-0-0] [lt=22] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:20.791260] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2195613}], scn_array=[cnt:1, {val:1736326279776878002, v:0}], curr_size=112, accum_size=112, time_cost=16382) [2025-01-08 00:51:20.797902] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C656-0-0] [lt=25][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193785711734, cache_obj->added_lc()=false, cache_obj->get_object_id()=13625, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:20.799503] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BECB-0-0] [lt=19] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:20.805062] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:491) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E55-0-0] [lt=71] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=148940998, MAX_GAP=900000000000, new_sync_scn={val:1736326278780077002, v:0}, new_replayable_scn={val:1736326278780077002, v:0}, new_readable_scn={val:1736326278631136004, v:0}, old_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326278012752001, replayable_scn:1736326278012752001, readable_scn:1736326278012752001, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}) [2025-01-08 00:51:20.807811] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=478, hit_cnt=4, read_cnt=4, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:20.808264] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2195669}, scn={val:1736326279776878002, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:20.808302] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:27, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326280806551, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2195725}, next_to_submit_scn_:{val:1736326279776878003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2195613}, end_lsn:{lsn:2195725}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2195725}, readable_log_tail:{lsn:2195725}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64909043, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326279776878002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326279776878002, v:0}, data_checksum:200933195, flag:49867}}, init_mode_version:0, accumulate_checksum:1273450875, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326279776878002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17949, miss_cnt:0, cache_read_size:2195725, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2195669}, scn_:{val:1736326279776878002, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326280661261, first_handle_ts_:1736326280808300, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad962c04150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:51:20.808833] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=84] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324475608151000) [2025-01-08 00:51:20.808862] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2195669}, scn_:{val:1736326279776878002, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326280661261, first_handle_ts_:1736326280808816, replay_cost_:44, retry_cost_:-1, read_log_buf_:0x2ad962c04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326280806551, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2195725}, next_to_submit_scn_:{val:1736326279776878003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2195613}, end_lsn:{lsn:2195725}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2195725}, readable_log_tail:{lsn:2195725}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64909043, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326279776878002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326279776878002, v:0}, data_checksum:200933195, flag:49867}}, init_mode_version:0, accumulate_checksum:1273450875, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326279776878002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17949, miss_cnt:0, cache_read_size:2195725, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=13, ret=0) [2025-01-08 00:51:20.809043] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=18] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:20.810394] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=41] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:20.811007] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=49] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:20.811532] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=39] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326275719333000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:20.812681] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EEE-0-0] [lt=20] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:20.815010] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=2361] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:20.815092] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:20.815156] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=145] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:20.820480] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8672, offset=2461542}, end_cursor:ObLogCursor{file_id=1, log_id=8673, offset=2461839}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a74050, len:4250, log_data_len:297, seq:8672, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:20.822076] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487A-0-0] [lt=29] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:20.822255] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487A-0-0] [lt=54] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:20.822403] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487A-0-0] [lt=146] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:20.822575] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E55-0-0] [lt=41] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:20.822651] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E55-0-0] [lt=51] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326279739822003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326279739822003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:20.822486] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=72] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:20.825074] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=2584] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:20.825324] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=237] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:20.825499] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=173] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:20.825563] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487A-0-0] [lt=57] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:20.825607] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487A-0-0] [lt=43] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:20.825640] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=25] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:20.825668] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=26] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:20.826038] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487A-0-0] [lt=364] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:20.826291] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487A-0-0] [lt=251] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:20.826337] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=37] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:20.826369] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=31] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:20.826420] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487A-0-0] [lt=46] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:20.826557] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487A-0-0] [lt=135] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:20.828688] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=32] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:20.828744] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=53] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:20.828816] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=66] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:20.828960] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=68] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:20.829017] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=56] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:20.829048] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=30] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:20.829078] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=26] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:20.829114] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487A-0-0] [lt=34] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:20.834353] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1736326280834349, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007661, add_bytes: 0 [2025-01-08 00:51:20.838750] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8673, offset=2461839}, end_cursor:ObLogCursor{file_id=1, log_id=8674, offset=2462136}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a78050, len:3953, log_data_len:297, seq:8673, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:20.839304] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8674, offset=2462136}, end_cursor:ObLogCursor{file_id=1, log_id=8675, offset=2462433}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a7c050, len:3656, log_data_len:297, seq:8674, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:20.857119] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=71] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:7835114706, tenant_id:1002}, cur_thread_count=7, cnt=1, new_thread_count=8) [2025-01-08 00:51:20.859680] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EEF-0-0] [lt=21] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [46262]OB_MALLOC COST TOO MUCH TIME, cost_time=111894, tenant_id=1001, label=CoStack, ctx_id=8, prio=1, size=515136, owner: ObMalloc, click_count: 1, time dist:[ALLOC_CHUNK_END=111891], seq:[0] [2025-01-08 00:51:20.860459] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A461B-0-0] [lt=88] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:20.860790] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A461B-0-0] [lt=314] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=3, avg_log_batch_cnt=0, total_group_log_size=357, avg_group_log_size=119) [2025-01-08 00:51:20.862083] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EF0-0-0] [lt=23] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:20.862397] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58509][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:20.862463] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:51:20.862499] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:20.863835] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58389][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=529] worker thread exit(tenant_->id()=1002, workers_.get_size()=6) [2025-01-08 00:51:20.864441] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT WRITE LOG COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=250431, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:20.864458] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=357) [2025-01-08 00:51:20.864477] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=177, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:20.864537] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=41, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:20.864872] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A8-0-0] [lt=68] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:20.864906] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A8-0-0] [lt=18] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326280694597000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326280694597000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:20.866039] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:20.866708] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=13] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:20.868068] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.869105] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=248344, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:20.869169] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=58] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=298, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:20.869290] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=101, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:20.875284] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=9] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:20.877498] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58461][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=31] worker thread exit(tenant_->id()=1, workers_.get_size()=6) [2025-01-08 00:51:20.880543] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C2A8-0-0] [lt=74] get number(ret=0, service_type_=0, range=1, base_id=1736326280879318000, start_id=1736326280879318000, end_id=1736326280879318001) [2025-01-08 00:51:20.883484] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=49] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:20.890355] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC863A-0-0] [lt=19][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1001, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:2174488}, new_end_lsn:{lsn:2174610}, old_advance_time_us:1736326279851141, new_ack_time_us:1736326280890351, advance delay(us):1039210}) [2025-01-08 00:51:20.892871] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC863B-0-0] [lt=33] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:20.892922] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC863B-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=3, avg_log_batch_cnt=0, total_group_log_size=356, avg_group_log_size=118) [2025-01-08 00:51:20.896843] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=50] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.896907] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.896935] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.897059] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:20.906474] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58510][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:20.906544] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58510][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8641-0-0] [lt=40] init point thread id with(&point=0x5572d50e4cc0, idx_=1166, point=[thread id=58510, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58510) [2025-01-08 00:51:20.906716] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47] worker thread created(tenant_->id()=1002, group_id_=1, token=6) [2025-01-08 00:51:20.907612] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FE-0-0] [lt=20] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:20.907653] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FE-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326280778701000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326280778701000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:20.914731] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=52] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18683088}, readable_log_tail:{lsn:18683088}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48421680, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:20.914911] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=177] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:20.914927] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2174854}, readable_log_tail:{lsn:2174854}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64929914, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:20.914979] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=51] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:20.914991] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] get_replay_process success(id={id:1002}, submitted_log_size=2208256, unsubmitted_log_size=2208256, replayed_log_size=2208256, unreplayed_log_size=0) [2025-01-08 00:51:20.915000] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] get_replay_process success(id={id:1003}, submitted_log_size=2195847, unsubmitted_log_size=2195847, replayed_log_size=2195847, unreplayed_log_size=0) [2025-01-08 00:51:20.915027] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:20.944398] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDED-0-0] [lt=19][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193785858229, cache_obj->added_lc()=false, cache_obj->get_object_id()=13626, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:20.960531] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D828-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=22, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:20.960563] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D828-0-0] [lt=31] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=219233, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:20.960574] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58465][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D828-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=193813, avg_log_submit_to_slide_time=219212) [2025-01-08 00:51:20.960781] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=58] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=478, hit_cnt=4, read_cnt=4, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:20.960966] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=132] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17951, miss_cnt:0, cache_read_size:2195969, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:20.962572] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8642-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=48, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:20.962597] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8642-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=216586, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:20.962607] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8642-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=94, avg_log_submit_to_flush_time=191284, avg_log_submit_to_slide_time=216491) [2025-01-08 00:51:20.962634] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=479, hit_cnt=4, read_cnt=4, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:20.962701] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18051, miss_cnt:0, cache_read_size:2208378, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:20.968266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:20.968971] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=95, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:20.969018] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=46] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=3793, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:20.969118] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=13] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:20.969294] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=174] [APPLY STAT CB EXECUTE TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=-1195, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:20.969307] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=13] [APPLY STAT CB TOTAL TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=2692, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:20.974700] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=45, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:20.974753] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=5, avg_log_batch_cnt=1, total_group_log_size=1538, avg_group_log_size=307) [2025-01-08 00:51:20.977552] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690E97-0-0] [lt=15] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.784684753417968750e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=892, Throttle Trigger(MB)=535, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.807327651615717912e+01) [2025-01-08 00:51:20.978773] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:20.978859] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=84] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=294485, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:20.983556] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=46] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:20.987865] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15044148903, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:20.987981] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:20.987997] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x2ad8881fc3b0, thread=0x2ad8c3fc1e30) [2025-01-08 00:51:20.988007] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15044148903, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:21.015716] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=84] no need to update min/max version(new_version={val:1736326279846662000, v:0}, min_version={val:1736326279846662000, v:0}, max_version={val:1736326279846662000, v:0}) [2025-01-08 00:51:21.019954] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=173] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:21.019988] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5452) [2025-01-08 00:51:21.068129] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1273968, replace_num=15728) [2025-01-08 00:51:21.068165] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:21.068526] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.068977] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=84] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:21.075939] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326280774792000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=301137, in_cluster_service=true, cluster_version={val:1736326279919756001, v:0}, min_cluster_version={val:1736326279919756001, v:0}, max_cluster_version={val:1736326279919756001, v:0}, get_cluster_version_err=0, cluster_version_delta=1156173, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36527, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=535572916, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:21.114796] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=15] ====== check clog disk timer task ====== [2025-01-08 00:51:21.114862] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=58] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:21.138790] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=41] ts response task statistics(alloc_count=60122, free_count=30060) [2025-01-08 00:51:21.138836] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=37] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=466, get_gts_with_stc_cnt=25, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=25, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:21.138912] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=74] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:21.138944] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=448, get_gts_with_stc_cnt=64, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=64, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:21.168599] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.181309] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D24EBAD49-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326281177892000, start_id=1736326281177892000, end_id=1736326281177892001) [2025-01-08 00:51:21.188127] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7837696238, tenant_id:1002}, cur_thread_count=8, cnt=-1, new_thread_count=7) [2025-01-08 00:51:21.188167] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:21.188262] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=93] try_wait return eagain(this=0x2ad8c0eb0e70, thread=0x2ad908915890) [2025-01-08 00:51:21.188275] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7837696238, tenant_id:1002}, cur_thread_count=8, cnt=-1, new_thread_count=7) [2025-01-08 00:51:21.189933] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:21.193346] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:21.254912] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=74] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:21.264751] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=217] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:21.265002] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:21.266471] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:21.269284] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:21.269551] WDIAG read_multi_row (ob_table_access_helper.h:238) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:21.271267] INFO delete_row (ob_table_access_helper.h:339) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326281269570) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326281269570) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:21.271345] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:21.271402] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:51:21.271452] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:51:21.271545] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=142] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:21.271794] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:21.271893] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.274792] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2208622}], scn_array=[cnt:1, {val:1736326281154552001, v:0}], curr_size=122, accum_size=488, time_cost=10094) [2025-01-08 00:51:21.275622] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] clock out of order(cur_ts=1736326281275619, cur_ts_=1736326281253870, delta=21749) [2025-01-08 00:51:21.276320] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2200309, this={dio_aligned_buf:{buf_write_offset:879, buf_padding_size:3217, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:24026, truncate_used_ts:5458}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2199552, buf_len=122, total_write_size=11318, total_write_size_after_dio=393216, ob_pwrite_used_ts=945724, count=93, each_pwrite_cost=10169) [2025-01-08 00:51:21.276378] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2196213}], scn_array=[cnt:1, {val:1736326280778701004, v:0}], curr_size=122, accum_size=722, time_cost=16896) [2025-01-08 00:51:21.277337] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=76] ====== check clog disk timer task ====== [2025-01-08 00:51:21.277383] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=38] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:21.300442] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C7BB606-0-0] [lt=94] get number(ret=0, service_type_=0, range=1, base_id=1736326281297509000, start_id=1736326281297509000, end_id=1736326281297509001) [2025-01-08 00:51:21.303240] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=52] start to print blacklist info [2025-01-08 00:51:21.303617] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=32] ls blacklist refresh finish(cost_time=49081) [2025-01-08 00:51:21.305379] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46384][T1_L0_G0][T1][YB42C0A89F8B-00062B2D170BBDAC-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326281305258000, start_id=1736326281305258000, end_id=1736326281305258001) [2025-01-08 00:51:21.315565] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58503][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=0] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:21.317793] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=48] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:21.317828] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-01-08 00:51:21.317839] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:21.317898] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=56] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:21.317913] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193785231399, flush_io_finished_ret_:0, flush_io_finished_round_:1944, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:21.319272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.320027] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.335876] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=52] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2208800}, scn={val:1736326281154552002, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:21.336175] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=66] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324475608151000) [2025-01-08 00:51:21.336275] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=97] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2208800}, scn_:{val:1736326281154552002, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326281434619, first_handle_ts_:1736326281336157, replay_cost_:115, retry_cost_:-1, read_log_buf_:0x2ad964804160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326281335572, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2208856}, next_to_submit_scn_:{val:1736326281154552003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2208744}, end_lsn:{lsn:2208856}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2208856}, readable_log_tail:{lsn:2208856}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64895912, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326281154552002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326281154552002, v:0}, data_checksum:244522390, flag:24718}}, init_mode_version:0, accumulate_checksum:2119773104, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326281154552002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18055, miss_cnt:0, cache_read_size:2208856, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=27, ret=0) [2025-01-08 00:51:21.357333] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1736326281357328, bytes: 875270471, bw: 0.234057 MB/s, add_ts: 1007707, add_bytes: 247318 [2025-01-08 00:51:21.365297] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB7-0-0] [lt=19] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:21.365337] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB7-0-0] [lt=36] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:21.372032] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.376319] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58] no need to update min/max version(new_version={val:1736326280977941000, v:0}, min_version={val:1736326280977941000, v:0}, max_version={val:1736326280977941000, v:0}) [2025-01-08 00:51:21.432420] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38C5-0-0] [lt=11][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18044, log_task={header:{begin_lsn:{lsn:2196213}, end_lsn:{lsn:2196335}, log_id:18044, min_scn:{val:1736326280778701004, v:0}, max_scn:{val:1736326280778701004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2196091}, prev_proposal_id:1, committed_end_lsn:{lsn:2196213}, data_checksum:940009244, accum_checksum:270770232, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326281259404, freeze_ts:1736326281259404, submit_ts:1736326281259418, flushed_ts:1736326281276942, gen_to_freeze cost time:0, gen_to_submit cost time:14, submit_to_flush cost time:17524}, fs_cb_begin_ts=1736326281432408, log_life_time=173004) [2025-01-08 00:51:21.438730] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=27] gc stale ls task succ [2025-01-08 00:51:21.459285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.459746] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.462580] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58512][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:21.463523] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:21.465378] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC864F-0-0] [lt=39] init point thread id with(&point=0x5572d50e4d40, idx_=1168, point=[thread id=58512, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58512) [2025-01-08 00:51:21.468119] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=36] ====== tenant freeze timer task ====== [2025-01-08 00:51:21.471729] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-01-08 00:51:21.471758] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=23] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71333860, tenant_id=1001) [2025-01-08 00:51:21.471794] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=26] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:21.471917] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:21.472397] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:21.472622] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=104][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.479875] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1736326281479110000, start_id=1736326281479110000, end_id=1736326281479110001) [2025-01-08 00:51:21.507392] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84325990}], scn_array=[cnt:1, {val:1736326281479110000, v:0}], curr_size=586, accum_size=3499, time_cost=27063) [2025-01-08 00:51:21.518403] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] no need to update min/max version(new_version={val:1736326281115250000, v:0}, min_version={val:1736326281115250000, v:0}, max_version={val:1736326281115250000, v:0}) [2025-01-08 00:51:21.519363] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.542158] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.544392] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.572825] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:21.572939] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=112][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.591952] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:21.592059] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=70] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48848, [delay/us]:prepare:12, schedule:94, submit:41, rt:73483, total:73632) [2025-01-08 00:51:21.592126] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:21.592279] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=135] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48842, [delay/us]: prepare:12, schedule:94, submit:41, device:73483, total:73632) [2025-01-08 00:51:21.592295] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:21.592336] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:17, bw:77334, [delay/us]:prepare:7, schedule:295, submit:42082, rt:3076, total:45462) [2025-01-08 00:51:21.592366] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:21.592406] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:77335, [delay/us]: prepare:7, schedule:295, submit:42082, device:3076, total:45462) [2025-01-08 00:51:21.592417] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:21.592439] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20351, [delay/us]:prepare:14, schedule:459, submit:1, rt:3367, total:3842) [2025-01-08 00:51:21.592487] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:39, bw:158744, [delay/us]:prepare:9, schedule:609, submit:810, rt:76329, total:77758) [2025-01-08 00:51:21.592540] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:21.592636] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20349, [delay/us]: prepare:14, schedule:459, submit:1, device:3367, total:3842) [2025-01-08 00:51:21.592682] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:39, avg_bw:158718, [delay/us]: prepare:9, schedule:609, submit:810, device:76329, total:77758) [2025-01-08 00:51:21.592695] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:21.592775] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4070, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:79654, total:79663) [2025-01-08 00:51:21.592811] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:21.592872] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=54] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4069, [delay/us]: prepare:0, schedule:0, submit:600000000, device:79654, total:79663) [2025-01-08 00:51:21.592893] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:21.592908] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:21.592937] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:21.592947] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:21.593009] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=58] [IO STATUS SERVER] net_in=312kB/s, net_out=264kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:21.643714] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=78] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326281643668], gts:1736326281642669000, latest_srr:[mts=1736326281643668]}) [2025-01-08 00:51:21.643793] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=44] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326281643761], gts:1736326281642669000, latest_srr:[mts=1736326281643761]}) [2025-01-08 00:51:21.644058] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326281644017], gts:1736326281643910000, latest_srr:[mts=1736326281644017]}) [2025-01-08 00:51:21.655678] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=33] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:21.664845] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=21] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:21.669672] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1003}, {id:1}]) [2025-01-08 00:51:21.669797] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:21.669876] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:21.669934] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=8] gc stale ls task succ [2025-01-08 00:51:21.670176] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F514-0-0] [lt=83] get number(ret=0, service_type_=0, range=1, base_id=1736326281665357000, start_id=1736326281665357000, end_id=1736326281665357001) [2025-01-08 00:51:21.672031] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:21.673067] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:21.673121] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:21.673283] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=159][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:21.673834] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.676852] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918542C-0-0] [lt=20][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:21.677074] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:21.677097] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-01-08 00:51:21.677105] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:21.677118] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:21.677131] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193785588740, flush_io_finished_ret_:0, flush_io_finished_round_:1940, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:21.677168] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:21.677183] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-01-08 00:51:21.677291] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=106] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:21.677305] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:21.677317] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193785588810, flush_io_finished_ret_:0, flush_io_finished_round_:3768, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:21.677966] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=441][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2209222}], scn_array=[cnt:1, {val:1736326281655245000, v:0}], curr_size=122, accum_size=1088, time_cost=11934) [2025-01-08 00:51:21.682101] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6124, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:21.682160] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=57] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3621) [2025-01-08 00:51:21.682188] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=83, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:21.682218] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4085) [2025-01-08 00:51:21.682346] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=72] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:21.682377] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=4164, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:21.682410] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:21.682426] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=180, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:21.686317] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870B8-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:21.686431] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:21.686447] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:21.687876] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58468][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6899-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=55444, avg_log_gen_to_submit_time=55515, avg_log_submit_to_flush_time=6252, avg_log_submit_to_slide_time=15480) [2025-01-08 00:51:21.687959] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3621, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:21.688098] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A928C-0-0] [lt=53] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1765) [2025-01-08 00:51:21.690157] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=247] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326281479110000, v:0}, new_max_version={val:1736326281479110000, v:0}, last_min_version={val:1736326281177892000, v:0}, last_max_version={val:1736326281177892000, v:0}, total_time_us=11393, delta_us=211044) [2025-01-08 00:51:21.690240] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=79] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326281479110000, v:0}, min_version={val:1736326281479110000, v:0}, max_version={val:1736326281479110000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:21.693325] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:21.693351] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:21.693460] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E8-0-0] [lt=53] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:21.693479] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E8-0-0] [lt=17] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:21.696991] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:21.697554] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA0-0-0] [lt=33][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:21.699006] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA0-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:21.701141] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A98-0-0] [lt=36] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:21.701187] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:21.701610] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:21.714927] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8659-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:21.715600] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:21.716222] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D05C-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:21.717606] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3C2-0-0] [lt=78][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:21.720526] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:21.720749] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58494][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=35] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:51:21.733358] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4744, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:21.733413] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1210) [2025-01-08 00:51:21.733440] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=246, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:21.733517] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=72, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:21.753450] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A99-0-0] [lt=22] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:21.755093] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58503][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:21.764134] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF55-0-0] [lt=45] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326281867795, keepalive_ts:-1}) [2025-01-08 00:51:21.764241] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF55-0-0] [lt=104] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326281867795, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:21.767620] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=17] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:21.767698] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:21.771599] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326277673832, last_first_register_time_us=1736324454612308, register_time_us=1736326277673832, curr_time_us=1736326281771596) [2025-01-08 00:51:21.771659] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326281771630, last_submit_register_req_time_us=1736326281771630, register_parent_reason=1) [2025-01-08 00:51:21.771675] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:21.773942] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085159-0-0] [lt=60] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:21.774296] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=82, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:21.774341] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:51:21.774361] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:21.774376] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:21.774446] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=42, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:21.774466] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:21.774479] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1098) [2025-01-08 00:51:21.774490] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:21.774370] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.774750] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/72:400504 disk:308K/72:400504 t1m2:48K/11:388310 t1m2g0:48K/11:388310 t1002m2:184K/45:456775 t1002m2g0:184K/45:456775 t1001m2:76K/16:250626 t1001m2g0:76K/16:250626 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:21.777005] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:21.777814] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A9A-0-0] [lt=33] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:21.779385] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC865A-0-0] [lt=21] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:21.779461] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC865A-0-0] [lt=75] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11265, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:21.779473] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC865A-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=104, avg_log_submit_to_flush_time=3976, avg_log_submit_to_slide_time=11321) [2025-01-08 00:51:21.779605] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1765, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:21.780881] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=41] failed to get result(ret=-4008) [2025-01-08 00:51:21.781074] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=70] check new jobs(ret=0, tenant_id=1002, is_oracle_tenant=false, job_infos=[{tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326232555425, start_date_:1736325329924481, last_date_:1736326229926861, this_date_:0, next_date_:1736327129924481, end_date_:64060560000000000, total_:6030537, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736410829924386, last_date_:0, this_date_:0, next_date_:1736410829924386, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:51:21.781285] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=207] check all tenants(ret=0) [2025-01-08 00:51:21.781944] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:21.782333] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC865B-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:21.782447] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC865B-0-0] [lt=113] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14857, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:21.782696] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=132] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=9709, avg_group_log_size=539) [2025-01-08 00:51:21.782724] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:21.785142] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=94] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=3630, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:21.785167] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10364) [2025-01-08 00:51:21.785184] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=2257, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:21.785261] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10831) [2025-01-08 00:51:21.785285] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=42, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:21.788723] INFO [LIB] stat (utility.h:1210) [58505][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D22-0-0] [lt=41] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=12, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:21.788744] INFO [LIB] stat (utility.h:1210) [58505][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D22-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=44614, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:21.788754] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58505][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D22-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=35559, avg_log_gen_to_submit_time=35643, avg_log_submit_to_flush_time=6236, avg_log_submit_to_slide_time=8971) [2025-01-08 00:51:21.788756] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5247413422, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:21.788938] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=93] recycle one thread(this=0x2ad8885abcf0, total=5, remain=4) [2025-01-08 00:51:21.788972] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=33] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5247413422, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:21.789057] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10364, hit_cnt=19, read_cnt=19, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:21.791302] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=53] ====== check clog disk timer task ====== [2025-01-08 00:51:21.791415] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=106] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:21.807419] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A9D-0-0] [lt=12] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:21.814685] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A462D-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1323) [2025-01-08 00:51:21.815997] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58507][T1002_TimerWK6][T1002][Y0-0000000000000000-0-0] [lt=0] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:21.820755] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A9E-0-0] [lt=30] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:21.830137] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=17] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:21.830177] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=28] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=180) [2025-01-08 00:51:21.830335] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [58502][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=0] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:21.830367] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [58502][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=24] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:21.830381] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [58502][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=11] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1) [2025-01-08 00:51:21.832605] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A9F-0-0] [lt=27] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:21.833438] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38CD-0-0] [lt=106] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1210) [2025-01-08 00:51:21.834541] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1736326281834538, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000189, add_bytes: 0 [2025-01-08 00:51:21.834797] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AA0-0-0] [lt=39] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:21.835657] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get_min_unreplayed_log_info(lsn={lsn:2209344}, scn={val:1736326281655245001, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326281683751, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2209344}, next_to_submit_scn_:{val:1736326281655245001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2209222}, end_lsn:{lsn:2209344}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2209466}, readable_log_tail:{lsn:2209466}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64895302, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326281655245000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326281655245000, v:0}, data_checksum:3348837161, flag:29314}}, init_mode_version:0, accumulate_checksum:3004572151, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326281655245000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18059, miss_cnt:0, cache_read_size:2209344, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:21.839427] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=85, stat_interval=1000000, avg (count/cost)=345, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:21.839469] INFO [LIB] stat (utility.h:1210) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=41] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=85, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:21.842450] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1769CA55-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1736326281839970000, start_id=1736326281839970000, end_id=1736326281839970001) [2025-01-08 00:51:21.842618] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1769CA55-0-0] [lt=28] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4085) [2025-01-08 00:51:21.842639] INFO [LIB] stat (utility.h:1210) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1769CA55-0-0] [lt=18] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:21.847547] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=56] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8675, offset=2462433}, end_cursor:ObLogCursor{file_id=1, log_id=8676, offset=2462738}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a80050, len:3359, log_data_len:305, seq:8675, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:21.848153] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E56-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:21.848211] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E56-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326281643910001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326281643910001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:21.857158] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C65A-0-0] [lt=47][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193786770989, cache_obj->added_lc()=false, cache_obj->get_object_id()=13627, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:21.872303] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46383][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C658-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193786786133, cache_obj->added_lc()=false, cache_obj->get_object_id()=13628, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:21.872321] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:21.873652] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:21.874709] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=139][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.879398] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:21.879880] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] no need to update min/max version(new_version={val:1736326281479110000, v:0}, min_version={val:1736326281479110000, v:0}, max_version={val:1736326281479110000, v:0}) [2025-01-08 00:51:21.881208] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58459][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=32] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-01-08 00:51:21.882518] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:21.885050] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AA1-0-0] [lt=35] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:21.891154] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A462F-0-0] [lt=77] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:21.891258] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A462F-0-0] [lt=91] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:51:21.894603] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3809, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:21.894630] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:51:21.894652] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=646, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:21.894714] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=74] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=74, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:21.896313] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.896475] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.896528] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.896556] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:21.901468] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8676, offset=2462738}, end_cursor:ObLogCursor{file_id=1, log_id=8677, offset=2463035}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a84050, len:3054, log_data_len:297, seq:8676, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:21.904665] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AA2-0-0] [lt=47] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:21.904777] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E56-0-0] [lt=534] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:21.904823] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E56-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326281643910001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326281643910001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:21.905714] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C660-0-0] [lt=113] get number(ret=0, service_type_=0, range=1, base_id=1736326281904723000, start_id=1736326281904723000, end_id=1736326281904723001) [2025-01-08 00:51:21.913761] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58513][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:21.913838] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58513][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C04-0-0] [lt=46] init point thread id with(&point=0x5572d50e4d80, idx_=1169, point=[thread id=58513, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58513) [2025-01-08 00:51:21.913969] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [58506][T1002_TimerWK5][T1002][YB42C0A89F89-00062B2D2868458E-0-0] [lt=0] InfoPoolResizeTask(cost_ts=9) [2025-01-08 00:51:21.914390] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:51:21.914809] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=15] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:21.915475] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:21.923365] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=156] get_max_applied_scn(scn={val:1736326281848053000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18685154}, palf_committed_end_scn_:{val:1736326281848053000, v:0}, last_check_scn_:{val:1736326281848053000, v:0}, max_applied_cb_scn_:{val:1736326281848053000, v:0}}) [2025-01-08 00:51:21.923546] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=175] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18685154}, readable_log_tail:{lsn:18685154}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48419614, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:21.923675] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=105] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:21.928291] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=48] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8677, offset=2463035}, end_cursor:ObLogCursor{file_id=1, log_id=8678, offset=2463332}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a88050, len:2757, log_data_len:297, seq:8677, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:21.932695] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8678, offset=2463332}, end_cursor:ObLogCursor{file_id=1, log_id=8679, offset=2463629}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a8c050, len:2460, log_data_len:297, seq:8678, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:21.932791] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38CF-0-0] [lt=64] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:21.932829] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38CF-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:21.940724] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FF-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:21.940774] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5FF-0-0] [lt=33] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326281783436001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326281783436001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:21.946034] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A9-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:21.946094] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2A9-0-0] [lt=46] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326281655245002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326281655245002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:21.960081] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=20] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:21.974657] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:21.978692] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=82] LogLoopThread round_cost_time(us)(round_cost_time=33) [2025-01-08 00:51:21.979027] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=103] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2739, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:21.979075] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=150, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:21.979125] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:21.980006] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] no need to update min/max version(new_version={val:1736326281479110000, v:0}, min_version={val:1736326281479110000, v:0}, max_version={val:1736326281479110000, v:0}) [2025-01-08 00:51:21.987506] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58505][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC4-0-0] [lt=11] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326282091929, keepalive_ts:-1}) [2025-01-08 00:51:21.987543] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58505][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC4-0-0] [lt=35] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326282091929, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:21.989324] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=67] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15049216120, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:21.989539] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=125] recycle one thread(this=0x2ad8881fc3b0, total=6, remain=5) [2025-01-08 00:51:21.989725] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=184] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15049216120, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:21.990977] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8662-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=22, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:21.991003] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8662-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=8936, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:21.991014] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8662-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=29, avg_log_submit_to_flush_time=4632, avg_log_submit_to_slide_time=8906) [2025-01-08 00:51:21.991122] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:21.991157] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18062, miss_cnt:0, cache_read_size:2209710, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:21.992770] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485378-0-0] [lt=36] MediumCheckTask(cost_ts=10) [2025-01-08 00:51:22.011329] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=43] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:22.023281] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=69, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:22.023334] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9474) [2025-01-08 00:51:22.025166] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326282020428000, start_id=1736326282020428001, end_id=1736326282020428002) [2025-01-08 00:51:22.037392] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9296-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:22.037424] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9296-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21838, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:22.037435] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9296-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=46, avg_log_submit_to_flush_time=4230, avg_log_submit_to_slide_time=21791) [2025-01-08 00:51:22.037504] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=198] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:22.037535] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17961, miss_cnt:0, cache_read_size:2197189, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:22.072595] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=78] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:22.074615] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:22.074796] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=95][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:22.080619] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [58506][T1002_TimerWK5][T1002][Y0-0000000000000000-0-0] [lt=21] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:22.080656] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [58506][T1002_TimerWK5][T1002][Y0-0000000000000000-0-0] [lt=26] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=1) [2025-01-08 00:51:22.082768] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=72] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=57, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:22.082799] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4487, avg_group_log_size=407) [2025-01-08 00:51:22.083138] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690E9D-0-0] [lt=46] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.784684753417968750e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=892, Throttle Trigger(MB)=535, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.807327651615717912e+01) [2025-01-08 00:51:22.086502] INFO [LIB] stat (utility.h:1210) [58468][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C05-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:22.086530] INFO [LIB] stat (utility.h:1210) [58468][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C05-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=8066, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:22.116519] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58509][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39C3-0-0] [lt=26] init point thread id with(&point=0x5572d50e4c80, idx_=1165, point=[thread id=58509, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58509) [2025-01-08 00:51:22.144489] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15] handle gts request statistics(total_rt=2249, total_cnt=282, avg_rt=7.946996466431095030e+00, avg_cnt=2.819999999999999929e+01) [2025-01-08 00:51:22.144570] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=66] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:22.174958] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=53] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:22.175152] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=192][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:22.189901] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7844830383, tenant_id:1002}, cur_thread_count=8, cnt=-1, new_thread_count=7) [2025-01-08 00:51:22.190078] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=108] recycle one thread(this=0x2ad8c0eb0e70, total=8, remain=7) [2025-01-08 00:51:22.190093] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7844830383, tenant_id:1002}, cur_thread_count=8, cnt=-1, new_thread_count=7) [2025-01-08 00:51:22.201529] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:22.201846] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:22.228035] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [58506][T1002_TimerWK5][T1002][Y0-0000000000000000-0-0] [lt=12] ====== traversal_flush timer task ====== [2025-01-08 00:51:22.228179] INFO [MDS] flush (mds_table_mgr.cpp:195) [58506][T1002_TimerWK5][T1002][YB42C0A89F89-00062B2D2868458F-0-0] [lt=29] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326281848053002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326290087570000, latest_log_ts:{val:1736326273510997001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.228432] INFO [MDS] flush (mds_table_mgr.cpp:172) [58506][T1002_TimerWK5][T1002][YB42C0A89F89-00062B2D2868458F-0-0] [lt=213] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277123056001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.228567] INFO [MDS] flush (mds_table_mgr.cpp:172) [58506][T1002_TimerWK5][T1002][YB42C0A89F89-00062B2D2868458F-0-0] [lt=87] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277013355003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.228730] INFO [MDS] flush (mds_table_mgr.cpp:172) [58506][T1002_TimerWK5][T1002][YB42C0A89F89-00062B2D2868458F-0-0] [lt=117] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277272403000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.228812] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [58506][T1002_TimerWK5][T1002][YB42C0A89F89-00062B2D2868458F-0-0] [lt=78] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:22.240457] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857A2-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193787154288, cache_obj->added_lc()=false, cache_obj->get_object_id()=13629, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:22.241602] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46386][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857A3-0-0] [lt=25][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:22.244692] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857A5-0-0] [lt=55][errcode=-4201] hashtable not init [2025-01-08 00:51:22.255331] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58503][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:22.265211] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:22.268157] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=129] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:22.268525] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58502][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:22.272984] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=104] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:22.275146] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:22.275186] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:22.316425] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:22.318005] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:22.318044] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-01-08 00:51:22.318054] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:22.318082] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:22.318100] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193786231786, flush_io_finished_ret_:0, flush_io_finished_round_:1945, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:22.318902] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.319423] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.333342] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=94] success to update min reserved snapshot(reserved_snapshot=1736324475719333000, duration=1800, snapshot_gc_ts=1736326275719333000) [2025-01-08 00:51:22.356327] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] ====== checkpoint timer task ====== [2025-01-08 00:51:22.357706] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=87] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:22.357793] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=85] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:22.357899] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=44] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326290087570000, latest_log_ts:{val:1736326273510997001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.358096] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=192] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:22.358133] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=29] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:22.358152] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] get rec log scn(service_type_=0, rec_log_ts={val:1736326273510997001, v:0}) [2025-01-08 00:51:22.358160] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:22.358175] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:22.358256] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:22.358323] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=62] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:22.358351] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:22.358388] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=35] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:22.359471] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:22.359630] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=158] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:22.359700] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=39] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277123056001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.360100] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=396] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:22.360124] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:22.360245] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=114] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:22.360381] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=131] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:22.360419] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2176318}, scn={val:1736326281848053002, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:22.360894] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=48] PNIO [ratelimit] time: 1736326282360891, bytes: 875552303, bw: 0.267822 MB/s, add_ts: 1003563, add_bytes: 281832 [2025-01-08 00:51:22.368071] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB8-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:22.368113] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB8-0-0] [lt=38] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:22.371447] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=70] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:22.374713] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=37] current monitor number(seq_=-1) [2025-01-08 00:51:22.375270] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=109][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:22.413709] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=25] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:22.423005] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=45] ====== traversal_flush timer task ====== [2025-01-08 00:51:22.423292] INFO [MDS] flush (mds_table_mgr.cpp:195) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EE3-0-0] [lt=27] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326282104848000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326289284252000, latest_log_ts:{val:1736326272625185001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.423399] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EE3-0-0] [lt=102] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:22.438873] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=26] gc stale ls task succ [2025-01-08 00:51:22.453421] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=21] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:22.453457] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=30] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:22.453483] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:22.453499] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=15] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:22.456915] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=9] Dumping All Cache Objs(alloc_obj_list.count()=132, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:4, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}]) [2025-01-08 00:51:22.457457] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=538] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:22.458154] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.458757] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.468096] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:22.468687] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58496][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=47] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:22.473268] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:22.475348] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:22.475973] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=105] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:22.519314] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.542264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=82] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.544624] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.555980] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [50389][T1002_PLSSer][T1001][YB42C0A89F89-00062B2D24884CAD-0-0] [lt=38] need check schema(new_schema_version=1736326191428712, cached_tenant_schema_version=1736326191428712, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT * FROM __all_ls_status WHERE tenant_id = ? ORDER BY tenant_id, ls_id) [2025-01-08 00:51:22.575437] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=116][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:22.577931] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:22.577984] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=51] try gc retain ctx [2025-01-08 00:51:22.577997] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] try to retry start_working [2025-01-08 00:51:22.578026] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] finish update min start scn(max_decided_scn={val:1736326282144239000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326272117597004, v:0}, keep_alive_scn_:{val:1736326277123056004, v:0}, update_ts_:1736326282577848}) [2025-01-08 00:51:22.579410] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=120] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326290087570000, latest_log_ts:{val:1736326273510997001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:22.579561] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=134] finish update min start scn(max_decided_scn={val:1736326281848053002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326272117597004, v:0}, keep_alive_scn_:{val:1736326277123056004, v:0}, update_ts_:1736326282579377}) [2025-01-08 00:51:22.580643] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277123056001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:22.580768] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=110] finish update min start scn(max_decided_scn={val:1736326281655245003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326275013027002, v:0}, keep_alive_scn_:{val:1736326280694597000, v:0}, update_ts_:1736326282580571}) [2025-01-08 00:51:22.582001] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=29] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277013355003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:22.582119] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=110] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:22.582156] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] get_min_unreplayed_log_info(lsn={lsn:2197311}, scn={val:1736326281783436004, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326282134971, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2197311}, next_to_submit_scn_:{val:1736326281783436004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2197189}, end_lsn:{lsn:2197311}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2197311}, readable_log_tail:{lsn:2197311}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64907457, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326281783436003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326281783436003, v:0}, data_checksum:2124447694, flag:40015}}, init_mode_version:0, accumulate_checksum:2786404142, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326281783436003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17962, miss_cnt:0, cache_read_size:2197311, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:22.582340] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=180] finish update min start scn(max_decided_scn={val:1736326281783436003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326274264437001, v:0}, keep_alive_scn_:{val:1736326279275750001, v:0}, update_ts_:1736326282581728}) [2025-01-08 00:51:22.583391] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277272403000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:22.604917] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:22.604990] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:6, bw:28306, [delay/us]:prepare:16, schedule:579, submit:26, rt:2471, total:3094) [2025-01-08 00:51:22.605072] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=74] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=1, iops=1, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:22.605236] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=56] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:6, avg_bw:28305, [delay/us]: prepare:16, schedule:579, submit:26, device:2471, total:3094) [2025-01-08 00:51:22.605257] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:22.605315] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:12, bw:56609, [delay/us]:prepare:7, schedule:220, submit:68, rt:2423, total:2719) [2025-01-08 00:51:22.605331] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:22.605444] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=89] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:12, avg_bw:56606, [delay/us]: prepare:7, schedule:220, submit:68, device:2423, total:2719) [2025-01-08 00:51:22.605461] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:22.605501] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16172, [delay/us]:prepare:17, schedule:68, submit:1, rt:3187, total:3274) [2025-01-08 00:51:22.605531] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:3, hold_mem:24576 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:23, bw:92993, [delay/us]:prepare:7, schedule:101, submit:94, rt:2607, total:2810) [2025-01-08 00:51:22.605542] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=8, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49997, free_result_cnt=49997, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:22.605587] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16174, [delay/us]: prepare:17, schedule:68, submit:1, device:3187, total:3274) [2025-01-08 00:51:22.605608] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:23, avg_bw:93005, [delay/us]: prepare:7, schedule:101, submit:94, device:2607, total:2810) [2025-01-08 00:51:22.605619] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:22.605639] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4043, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:609, total:619) [2025-01-08 00:51:22.605722] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:22.605777] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=49] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4043, [delay/us]: prepare:0, schedule:0, submit:600000000, device:609, total:619) [2025-01-08 00:51:22.605796] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:22.605813] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=1, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 1 [2025-01-08 00:51:22.605833] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:22.605847] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:22.605858] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=201kB/s, net_out=173kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:22.792866] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17] submit log success(service_type_=0, last_id=1736326299254996000, limited_id=1736326299254996000) [2025-01-08 00:51:22.793023] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=155] submit log with lock success(service_type_=0, last_id=1736326299254996000, limited_id=1736326299254996000) [2025-01-08 00:51:22.794577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=208][errcode=0] task scheduled too delay(scheduled_delay=288107, expected_run_ts_us_=1736326282506463, ObClockGenerator::getRealClock()=1736326282794573, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:22.506, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:22.795447] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=164] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:22.796011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] task scheduled too delay(scheduled_delay=246652, expected_run_ts_us_=1736326282549355, ObClockGenerator::getRealClock()=1736326282796009, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:22.549, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:22.796622] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1003}, {id:1}]) [2025-01-08 00:51:22.796687] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326272266072, end_time=1736326282796685, dag_status={}) [2025-01-08 00:51:22.796707] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:22.796720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:22.796728] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:22.796735] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:22.796742] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:22.796749] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:22.796756] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:22.796763] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:22.796770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:22.796778] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796789] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796798] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796806] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796819] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796831] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796841] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796850] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796858] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796866] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796874] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796882] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796890] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796899] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796907] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796915] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796922] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796930] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796939] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796947] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796955] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796963] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796970] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796978] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796986] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.796994] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797002] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797010] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797018] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797026] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797035] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797043] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797050] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797058] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797066] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797074] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797082] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797089] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797097] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797105] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797114] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797122] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797130] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797138] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797146] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797154] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797162] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797169] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797244] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797253] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797261] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797269] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797277] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797285] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797292] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797300] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797308] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797316] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797333] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797340] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797348] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797356] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797364] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797371] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797379] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797387] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797395] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797403] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797411] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797419] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797413] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46102][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=31] expand thread count(this=0x5572d527f900, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:22.797427] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797435] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:22.797445] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:22.797452] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:22.797470] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=17] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:22.797475] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:22.797481] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:22.797486] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:22.797491] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:22.797496] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:22.797501] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:22.797506] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:22.797538] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=32] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:22.797580] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:22.797611] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=13] gc stale ls task succ [2025-01-08 00:51:22.797670] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=12] gc stale ls task succ [2025-01-08 00:51:22.797991] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:22.798206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50378][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=71][errcode=0] task scheduled too delay(scheduled_delay=290537, expected_run_ts_us_=1736326282507666, ObClockGenerator::getRealClock()=1736326282798204, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05d30, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8efe559b0, time_wheel:0x2ad8cf1f7990, thread_pool:0x2ad8cf1f7170, is_running:1, time_interval:1.00s, expected_run_time:00:51:22.507, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:22.800814] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:22.800846] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:22.802480] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=239][errcode=-4016] task executed too delay(delay_time=73612, *p_task_={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:0, time_interval:250.00ms, expected_run_time:00:51:22.908, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:22.802595] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918542D-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:22.806397] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84646-0-0] [lt=33] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2025-01-08 00:51:22.808054] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84646-0-0] [lt=30] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2025-01-08 00:51:22.808046] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D40-0-0] [lt=15] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:22.808089] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1010) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84646-0-0] [lt=32] no transfer partition job [2025-01-08 00:51:22.808107] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1040) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84646-0-0] [lt=12] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer=true) [2025-01-08 00:51:22.808947] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=13] ====== tenant freeze timer task ====== [2025-01-08 00:51:22.810533] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58502][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=217] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:22.810616] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58502][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:22.811097] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=61229, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:22.811118] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43281182}], scn_array=[cnt:1, {val:1736326282223734000, v:0}], curr_size=589, accum_size=6873, time_cost=584664) [2025-01-08 00:51:22.811257] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=679, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:22.811853] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:284) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84646-0-0] [lt=15] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-01-08 00:51:22.811909] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:493) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84646-0-0] [lt=36] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2025-01-08 00:51:22.811957] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:157) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84646-0-0] [lt=36] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=3, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1003}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone3", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1736324440080711, last_statistic_bg_stat_time=1736326192603087, last_statistic_schema_version=1736326192037600, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-01-08 00:51:22.813022] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58515][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=0] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:22.813697] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46102][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=22] expand thread count(this=0x5572d527f900, max_thread_cnt=128, cur_thread_count=5, inc_cnt=1, queue_size=5) [2025-01-08 00:51:22.821399] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58516][TimerWK5][T0][Y0-0000000000000000-0-0] [lt=0] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6718582784, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:22.821843] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA1-0-0] [lt=33][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:22.822451] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=92537, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:22.822473] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84329102}], scn_array=[cnt:1, {val:1736326282144239000, v:0}], curr_size=122, accum_size=2526, time_cost=539413) [2025-01-08 00:51:22.822502] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=64, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:22.822527] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18685520}], scn_array=[cnt:1, {val:1736326282144239001, v:0}], curr_size=122, accum_size=1033, time_cost=545033) [2025-01-08 00:51:22.822580] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=93520, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:22.822594] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2176318}], scn_array=[cnt:1, {val:1736326282144239001, v:0}], curr_size=122, accum_size=732, time_cost=544921) [2025-01-08 00:51:22.822610] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=376, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:22.822619] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=118912, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:22.822638] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2197311}], scn_array=[cnt:1, {val:1736326281783436004, v:0}], curr_size=122, accum_size=610, time_cost=581998) [2025-01-08 00:51:22.822659] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=1706, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:22.822729] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=116] [PALF STAT FLUSH CB COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:22.822780] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:22.822906] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:22.823180] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=48, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:22.823357] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:164K/39:129279 disk:164K/39:129279 t1m2:24K/5:18572 t1m2g0:24K/5:18572 t1002m2:92K/23:108888 t1002m2g0:92K/23:108888 t1001m2:48K/11:222235 t1001m2g0:48K/11:222235 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:22.823976] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [58516][TimerWK5][T0][Y0-0000000000000000-0-0] [lt=30] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=849339, clean_num=31457) [2025-01-08 00:51:22.824062] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=148] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:6, threads_idle_time:15117665722, tenant_id:500}, cur_thread_count=6, cnt=1, new_thread_count=7) [2025-01-08 00:51:22.825166] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDC-0-0] [lt=79] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:22.825188] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDC-0-0] [lt=16] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=24) [2025-01-08 00:51:22.825397] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8668-0-0] [lt=110] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:22.825411] INFO [LIB] stat (utility.h:1210) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8668-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=113600, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:22.825420] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8668-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=17881, log_task={header:{begin_lsn:{lsn:2176318}, end_lsn:{lsn:2176440}, log_id:17881, min_scn:{val:1736326282144239001, v:0}, max_scn:{val:1736326282144239001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2176318}, data_checksum:2222604892, accum_checksum:429818404, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326282277495, freeze_ts:1736326282277499, submit_ts:1736326282277564, flushed_ts:1736326282822723, gen_to_freeze cost time:4, gen_to_submit cost time:69, submit_to_flush cost time:545159}, fs_cb_begin_ts=1736326282825382, log_life_time=547887) [2025-01-08 00:51:22.825498] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58465][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A929D-0-0] [lt=28][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18053, log_task={header:{begin_lsn:{lsn:2197311}, end_lsn:{lsn:2197433}, log_id:18053, min_scn:{val:1736326281783436004, v:0}, max_scn:{val:1736326281783436004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2197189}, prev_proposal_id:1, committed_end_lsn:{lsn:2197311}, data_checksum:940009244, accum_checksum:3468287762, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326282232531, freeze_ts:1736326282232531, submit_ts:1736326282232539, flushed_ts:1736326282822745, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:590206}, fs_cb_begin_ts=1736326282825487, log_life_time=592956) [2025-01-08 00:51:22.826936] INFO pn_ratelimit (group.c:653) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=59] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:22.826998] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=649809920, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=423931904, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:22.827069] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=32] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:22.827272] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=120] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2842, drift_size_=0, pre_drift_size=-6291456, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:22.827344] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1118584832, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2102640640, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:22.827400] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:22.827414] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:22.828507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] task scheduled too delay(scheduled_delay=145049, expected_run_ts_us_=1736326282683455, ObClockGenerator::getRealClock()=1736326282828505, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:22.683, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:22.828562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] task scheduled too delay(scheduled_delay=126262, expected_run_ts_us_=1736326282702299, ObClockGenerator::getRealClock()=1736326282828561, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:22.702, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:22.829709] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=54] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:3, threads_idle_time:15117666072, tenant_id:500}, cur_thread_count=6, cnt=1, new_thread_count=7) [2025-01-08 00:51:22.829732] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=20] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:22.829890] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58468][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C07-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=44733, log_task={header:{begin_lsn:{lsn:84329102}, end_lsn:{lsn:84329224}, log_id:44733, min_scn:{val:1736326282144239000, v:0}, max_scn:{val:1736326282144239000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84329102}, data_checksum:667309872, accum_checksum:2657050994, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326282282931, freeze_ts:1736326282282936, submit_ts:1736326282282995, flushed_ts:1736326282822826, gen_to_freeze cost time:5, gen_to_submit cost time:64, submit_to_flush cost time:539831}, fs_cb_begin_ts=1736326282829863, log_life_time=546932) [2025-01-08 00:51:22.830038] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A929B-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=33, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:22.830052] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A929B-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=114340, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:22.830060] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A929B-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=19517, log_task={header:{begin_lsn:{lsn:18685520}, end_lsn:{lsn:18685642}, log_id:19517, min_scn:{val:1736326282144239001, v:0}, max_scn:{val:1736326282144239001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18685520}, data_checksum:667309872, accum_checksum:1189422749, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326282277302, freeze_ts:1736326282277318, submit_ts:1736326282277466, flushed_ts:1736326282822702, gen_to_freeze cost time:16, gen_to_submit cost time:164, submit_to_flush cost time:545236}, fs_cb_begin_ts=1736326282830028, log_life_time=552726) [2025-01-08 00:51:22.834017] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:22.834464] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=82] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:22.836977] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:22.837047] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=59] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=87) [2025-01-08 00:51:22.837744] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=21] update id meta(ret=0, service_type=0, limited_id=1736326299254996000, latest_scn={val:1736326282603912001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299254996000, latest_log_ts:{val:1736326282603912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:22.837799] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=54] handle submit callback(service_type_=0, success=true, used_ts=44938, log_ts={val:1736326282603912001, v:0}, limited_id=1736326299254996000) [2025-01-08 00:51:22.837813] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=13] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:22.837822] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:22.837895] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=66][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326282279763000, v:0}, log_ts:{val:1736326282279763000, v:0}, lsn:{lsn:84329280}, submit_ts:1736326282283774}, this:0x2ad89c0a3bd8, is_inited_:true, trans_id:{txid:27999}, ls_id:{id:1}, ctx:0x2ad89c0a01d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=554118, ctx_lock_wait_time=1) [2025-01-08 00:51:22.838180] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=39] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324475719333000) [2025-01-08 00:51:22.838253] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=16] update reserved snapshot finished(cost_ts=504913, reserved_snapshot=1736324475719333000) [2025-01-08 00:51:22.840513] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:22.840568] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585457-0-0] [lt=64] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:22.840589] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585457-0-0] [lt=13] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:22.841155] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA1-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:22.841645] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1914) [2025-01-08 00:51:22.841937] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1736326282841933, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007395, add_bytes: 0 [2025-01-08 00:51:22.844047] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:22.845293] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58497][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=15] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:51:22.849070] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=48] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8679, offset=2463629}, end_cursor:ObLogCursor{file_id=1, log_id=8680, offset=2463926}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a90050, len:2163, log_data_len:297, seq:8679, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:22.849261] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:22.849384] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326281848053002, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:22.849417] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:22.849441] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:22.850171] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:22.850271] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=99] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:22.850299] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4015] clock out of order(cur_ts=1736326282850297, cur_ts_=1736326282603912, delta=246385) [2025-01-08 00:51:22.850366] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=51] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277013355003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.850460] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=91] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:22.850479] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:22.850498] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:22.850515] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:22.850531] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2209710}, scn={val:1736326281655245003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:22.850648] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F8B-00062B2D18EC335F-0-0] [lt=35][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=213744, send_timestamp=1736326282636901, peer="192.168.159.139:45062", sz=265) [2025-01-08 00:51:22.852223] INFO [LIB] stat (utility.h:1210) [58505][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39C4-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:22.852246] INFO [LIB] stat (utility.h:1210) [58505][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39C4-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=71009, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:22.852255] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58505][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39C4-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=32737, log_task={header:{begin_lsn:{lsn:43281182}, end_lsn:{lsn:43281771}, log_id:32737, min_scn:{val:1736326282223734000, v:0}, max_scn:{val:1736326282223734000, v:0}, data_len:533, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43281182}, data_checksum:2108407325, accum_checksum:1365549570, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326282225039, freeze_ts:1736326282225124, submit_ts:1736326282225372, flushed_ts:1736326282823117, gen_to_freeze cost time:85, gen_to_submit cost time:333, submit_to_flush cost time:597745}, fs_cb_begin_ts=1736326282852166, log_life_time=627127) [2025-01-08 00:51:22.852412] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326282223734000, v:0}, log_ts:{val:1736326282223734000, v:0}, lsn:{lsn:43281238}, submit_ts:1736326282225029}, this:0x2ad8fb7086d8, is_inited_:true, trans_id:{txid:66868}, ls_id:{id:1}, ctx:0x2ad8fb704cd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=627378, ctx_lock_wait_time=1) [2025-01-08 00:51:22.861647] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58402][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4635-0-0] [lt=57][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18155, log_task={header:{begin_lsn:{lsn:2209710}, end_lsn:{lsn:2209832}, log_id:18155, min_scn:{val:1736326281655245004, v:0}, max_scn:{val:1736326281655245004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2209588}, prev_proposal_id:1, committed_end_lsn:{lsn:2209710}, data_checksum:1715074492, accum_checksum:659974768, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326282089017, freeze_ts:1736326282089017, submit_ts:1736326282089025, flushed_ts:1736326282093659, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:4634}, fs_cb_begin_ts=1736326282861621, log_life_time=772604) [2025-01-08 00:51:22.862140] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=131] ====== traversal_flush timer task ====== [2025-01-08 00:51:22.862185] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C661-0-0] [lt=29][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193787776018, cache_obj->added_lc()=false, cache_obj->get_object_id()=13630, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:22.862468] INFO [MDS] flush (mds_table_mgr.cpp:195) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857C5-0-0] [lt=20] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326282603912001, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299254996000, latest_log_ts:{val:1736326282603912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.862566] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857C5-0-0] [lt=90] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:22.864395] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=40] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=101, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:22.864422] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:22.864617] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=56][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326282226897000, v:0}, log_ts:{val:1736326282226897000, v:0}, lsn:{lsn:43281827}, submit_ts:1736326282227581}, this:0x2ad8fb6c4ed8, is_inited_:true, trans_id:{txid:66869}, ls_id:{id:1}, ctx:0x2ad8fb6c14d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=637032, ctx_lock_wait_time=1) [2025-01-08 00:51:22.864739] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=173] Slow write(duration=10874, write_len=4096) [2025-01-08 00:51:22.864762] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=22] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8680, offset=2463926}, end_cursor:ObLogCursor{file_id=1, log_id=8681, offset=2464223}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a94050, len:1866, log_data_len:297, seq:8680, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:22.864950] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=42] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326281655245003, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:22.865013] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:22.865037] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:22.865773] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:22.865811] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=38] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:22.865856] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326277272403000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326278780077002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:22.866123] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=263] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:22.866141] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:22.866159] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:22.866176] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:22.866270] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=86] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2197433}, scn={val:1736326281783436004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:22.867691] WDIAG [PALF] runTimerTask (log_updater.cpp:109) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] update_palf cost too much time(ret=0, cost_time_us=599950, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:22.870067] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870BE-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:22.870112] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:22.870124] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:22.871933] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:801) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852A3-0-0] [lt=40] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, location:{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, location:{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}]) [2025-01-08 00:51:22.875165] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D05F-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:22.875602] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=16] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8681, offset=2464223}, end_cursor:ObLogCursor{file_id=1, log_id=8682, offset=2464520}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a98050, len:1569, log_data_len:297, seq:8681, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:22.877428] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326281783436004, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:22.877520] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:22.877548] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:22.877560] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:22.879634] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3C5-0-0] [lt=101][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:22.892287] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB93-0-0] [lt=25][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193787806118, cache_obj->added_lc()=false, cache_obj->get_object_id()=13631, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:22.893129] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] BlockGCTimerTask success(ret=0, cost_time_us=24, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:22.896558] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=7650, avg_group_log_size=637) [2025-01-08 00:51:22.896605] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:22.897332] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.897385] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.897422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.897447] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:22.899125] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4637-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:22.899170] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4637-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=4, avg_log_batch_cnt=0, total_group_log_size=488, avg_group_log_size=122) [2025-01-08 00:51:22.899249] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4637-0-0] [lt=76] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=610) [2025-01-08 00:51:22.898444] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1001, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:17881, max_lsn:2176440, max_scn:{val:1736326282144239001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2176440}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2176318}, last_submit_end_lsn:{lsn:2176440}, last_submit_log_id:17881, last_submit_log_pid:1, max_flushed_lsn:{lsn:2176318}, max_flushed_end_lsn:{lsn:2176440}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2176440}, last_slide_log_id:17881, last_slide_scn:{val:1736326282144239001, v:0}, last_slide_lsn:{lsn:2176318}, last_slide_end_lsn:{lsn:2176440}, last_slide_log_pid:1, last_slide_log_accum_checksum:429818404, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924f6a530}, [StateMgr]={this:0x2ad924f6e830, self:"192.168.159.137:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:17882, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324447115984, palf_role_change_cb:0x2ad924f752f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924f6bc70}, [ModeMgr]={palf_id:1001, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324447762546, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436756804001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/meta", dir_fd:177, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924f70a38, flashback_version:0}, log_storage:{log_tail:{lsn:2176440}, readable_log_tail:{lsn:2176440}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64928328, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1736324447114985, this:0x2ad924f6ef70}, [Reconfirm]={palf_id:1001, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:1"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324447117136, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924f6e9b0}) [2025-01-08 00:51:22.899396] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=945] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326282822724, last_flushed_end_lsn:{lsn:2176440}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326282852686, last_flushed_end_lsn:{lsn:2176440}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326282825346, last_flushed_end_lsn:{lsn:2176440}}]) [2025-01-08 00:51:22.899420] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] current log_task status(palf_id=1001, self="192.168.159.137:2882", log_id=17882, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326282822724, last_flushed_end_lsn:{lsn:2176440}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326282852686, last_flushed_end_lsn:{lsn:2176440}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326282825346, last_flushed_end_lsn:{lsn:2176440}}]) [2025-01-08 00:51:22.899929] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:22.900544] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=7650) [2025-01-08 00:51:22.900588] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=7650) [2025-01-08 00:51:22.902888] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58518][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:22.902997] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58518][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38D7-0-0] [lt=78] init point thread id with(&point=0x5572d50e4ec0, idx_=1174, point=[thread id=58518, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58518) [2025-01-08 00:51:22.903043] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58518][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38D7-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=610) [2025-01-08 00:51:22.905746] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=732) [2025-01-08 00:51:22.908269] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58474][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D2E-0-0] [lt=42] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=49887, avg_log_gen_to_submit_time=49972, avg_log_submit_to_flush_time=54020, avg_log_submit_to_slide_time=63228) [2025-01-08 00:51:22.908548] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=7650, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:22.909346] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:22.909969] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=46] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:22.911502] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=4820, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:22.911530] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2209954}], scn_array=[cnt:1, {val:1736326282603912001, v:0}], curr_size=122, accum_size=488, time_cost=11172) [2025-01-08 00:51:22.911553] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=488) [2025-01-08 00:51:22.911582] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=441, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:22.911668] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=142] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=126, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:22.911719] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=6, avg_log_batch_cnt=1, total_group_log_size=1155, avg_group_log_size=192) [2025-01-08 00:51:22.911737] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1155) [2025-01-08 00:51:22.911751] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:22.911804] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=79, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:22.911834] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=6, avg_log_batch_cnt=1, total_group_log_size=732, avg_group_log_size=122) [2025-01-08 00:51:22.911850] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=732) [2025-01-08 00:51:22.911861] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:22.911986] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=55, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:22.913565] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=6] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=854) [2025-01-08 00:51:22.913643] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1155) [2025-01-08 00:51:22.913701] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=62] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=732) [2025-01-08 00:51:22.913816] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=101] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1155) [2025-01-08 00:51:22.921071] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:22.921720] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58471][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=34] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-01-08 00:51:22.921739] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=125] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=610, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:22.922616] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A92A0-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=215, avg_log_submit_to_flush_time=92766, avg_log_submit_to_slide_time=96915) [2025-01-08 00:51:22.922753] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1155, hit_cnt=6, read_cnt=6, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:22.938297] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56] no need to update min/max version(new_version={val:1736326281904723000, v:0}, min_version={val:1736326281904723000, v:0}, max_version={val:1736326281904723000, v:0}) [2025-01-08 00:51:22.954054] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1736326282104848000, v:0}, base_version={val:1736326281860779000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326282104848000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326282954027, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326282104848000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326283004373, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326282104848000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326282907400, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:22.956330] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=100] submit log success(service_type_=0, last_id=1736326299284252000, limited_id=1736326299284252000) [2025-01-08 00:51:22.956368] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] submit log with lock success(service_type_=0, last_id=1736326299284252000, limited_id=1736326299284252000) [2025-01-08 00:51:22.956380] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326282955407000, start_id=1736326282955407000, end_id=1736326282955407001) [2025-01-08 00:51:22.959592] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=39] update id meta(ret=0, service_type=0, limited_id=1736326299284252000, latest_scn={val:1736326282955407001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299284252000, latest_log_ts:{val:1736326282955407001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:22.959660] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=67] handle submit callback(service_type_=0, success=true, used_ts=3332, log_ts={val:1736326282955407001, v:0}, limited_id=1736326299284252000) [2025-01-08 00:51:22.959784] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=123] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:22.959795] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:22.963358] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326282963100000, start_id=1736326282963100000, end_id=1736326282963100001) [2025-01-08 00:51:22.963934] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326282104848000, v:0}, new_max_version={val:1736326282104848000, v:0}, last_min_version={val:1736326281860779000, v:0}, last_max_version={val:1736326281860779000, v:0}, total_time_us=9775, delta_us=859083) [2025-01-08 00:51:22.964015] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=77] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326282104848000, v:0}, min_version={val:1736326282104848000, v:0}, max_version={val:1736326282104848000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:22.997008] INFO [LIB] stat (utility.h:1210) [58518][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4639-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=30, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:22.997045] INFO [LIB] stat (utility.h:1210) [58518][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4639-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=302697, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:22.997056] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58518][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4639-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=3, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=22, avg_log_submit_to_flush_time=6581, avg_log_submit_to_slide_time=302674) [2025-01-08 00:51:22.996998] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2654) [2025-01-08 00:51:22.997081] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=79] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:22.997257] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=148] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=366, hit_cnt=3, read_cnt=3, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:22.997293] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18065, miss_cnt:0, cache_read_size:2210076, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:22.999094] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3362) [2025-01-08 00:51:22.999195] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3362) [2025-01-08 00:51:23.000833] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.002072] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38D9-0-0] [lt=25] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:23.002228] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58512][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38D9-0-0] [lt=93] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=5, avg_log_batch_cnt=0, total_group_log_size=610, avg_group_log_size=122) [2025-01-08 00:51:23.002992] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5249961280, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:23.003025] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:23.003037] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a0110) [2025-01-08 00:51:23.003045] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5249961280, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:23.003056] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15053736898, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:23.003067] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:23.003074] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=7] try_wait return eagain(this=0x2ad8881fc3b0, thread=0x2ad8aedf1960) [2025-01-08 00:51:23.003079] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=5] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15053736898, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:23.006240] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58358][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68A1-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=67683, avg_log_gen_to_submit_time=67757, avg_log_submit_to_flush_time=71633, avg_log_submit_to_slide_time=74839) [2025-01-08 00:51:23.006335] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3362, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:23.006792] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=32] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:51:23.006833] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=18] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=1) [2025-01-08 00:51:23.006845] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:51:23.006863] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=14] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:51:23.007636] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58519][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:23.007788] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=106][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:23.008883] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=40][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:23.008916] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=16][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:23.009329] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:51:23.013933] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=110395, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:23.013998] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=91, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:23.014026] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=44, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:23.028153] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58515][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:23.030276] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:23.042786] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=11] runTimer to refresh locality_info(ret=0) [2025-01-08 00:51:23.045570] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:346) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=18] set_server_locality_array success(server_locality_array=[{inited:true, addr:"192.168.159.137:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322580423131, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.138:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581270010, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.139:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581175973, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-01-08 00:51:23.045699] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=63] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-01-08 00:51:23.045850] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=148] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-01-08 00:51:23.045872] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=13] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1001, primary_zone=RANDOM) [2025-01-08 00:51:23.045878] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] add_locality_zone success(locality_zone={tenant_id:1001, region_priority:18446744073709551615}) [2025-01-08 00:51:23.045890] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1002, primary_zone=RANDOM) [2025-01-08 00:51:23.045895] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5] add_locality_zone success(locality_zone={tenant_id:1002, region_priority:18446744073709551615}) [2025-01-08 00:51:23.045904] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1001, region_priority:18446744073709551615}, {tenant_id:1002, region_priority:18446744073709551615}]}) [2025-01-08 00:51:23.047066] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852A4-0-0] [lt=122] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326278364497, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:23.047123] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852A4-0-0] [lt=55] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326278366414, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:23.047147] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852A4-0-0] [lt=22] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, renew_time:1736326278367965, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, renew_time:1736326278367967, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, renew_time:1736326278367969, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326278367961, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:23.097225] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4015] clock out of order(cur_ts=1736326283097219, cur_ts_=1736326283062476, delta=34743) [2025-01-08 00:51:23.170459] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=121] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=77, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:23.170458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50522][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=91][errcode=0] task scheduled too delay(scheduled_delay=225142, expected_run_ts_us_=1736326282945309, ObClockGenerator::getRealClock()=1736326283170453, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd190, caller:mds_tenant_service.cpp:mtl_start:135, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8f0dfd470, time_wheel:0x2ad8dacdf990, thread_pool:0x2ad8dacdf170, is_running:1, time_interval:15.00s, expected_run_time:00:51:22.945, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:23.170513] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4394) [2025-01-08 00:51:23.170556] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50522][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=58][errcode=0] task scheduled too delay(scheduled_delay=225248, expected_run_ts_us_=1736326282945306, ObClockGenerator::getRealClock()=1736326283170554, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfcdb0, caller:mds_tenant_service.cpp:mtl_start:125, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8f0dfd090, time_wheel:0x2ad8dacdf990, thread_pool:0x2ad8dacdf170, is_running:1, time_interval:3.00s, expected_run_time:00:51:22.945, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:23.170761] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.171968] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BA3-0-0] [lt=14] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:23.172135] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BA5-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:23.175168] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] ====== check clog disk timer task ====== [2025-01-08 00:51:23.175188] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=17] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:23.176584] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Slow write(duration=78281, write_len=4096) [2025-01-08 00:51:23.176620] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8682, offset=2464520}, end_cursor:ObLogCursor{file_id=1, log_id=8683, offset=2464825}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894a9c050, len:1272, log_data_len:305, seq:8682, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:23.177997] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E57-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:23.178056] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E57-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326282603912000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326282603912000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:23.181089] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AD6F-0-0] [lt=95] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.840151977539062500e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=392, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.431660989222258422e+02) [2025-01-08 00:51:23.198583] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=74, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:23.198625] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=6, avg_log_batch_cnt=1, total_group_log_size=2252, avg_group_log_size=375) [2025-01-08 00:51:23.231760] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15120032789, tenant_id:500}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:23.231836] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:23.231856] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] try_wait return eagain(this=0x5572d527f900, thread=0x2ad8d4205b00) [2025-01-08 00:51:23.231868] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15120032789, tenant_id:500}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:23.231884] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7850792830, tenant_id:1002}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:23.232056] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=164][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:23.232078] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=21] try_wait return eagain(this=0x2ad8c0eb0e70, thread=0x2ad9349e7e00) [2025-01-08 00:51:23.232132] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=52] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7850792830, tenant_id:1002}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:23.232514] INFO [LIB] stat (utility.h:1210) [58490][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38DD-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:23.232541] INFO [LIB] stat (utility.h:1210) [58490][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38DD-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=151649, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:23.232551] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58490][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38DD-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=27, avg_log_submit_to_flush_time=120170, avg_log_submit_to_slide_time=151621) [2025-01-08 00:51:23.232697] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=610, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:23.232749] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17966, miss_cnt:0, cache_read_size:2197799, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:23.233651] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58516][TimerWK5][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:23.233726] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46100][TimerWK2][T0][YB42C0A89F89-00062B2D17E85440-0-0] [lt=12] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:23.234123] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58515][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=123] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6731165696, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:23.234267] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4015] clock out of order(cur_ts=1736326283234265, cur_ts_=1736326283198396, delta=35869) [2025-01-08 00:51:23.234722] WDIAG [PALF] is_prev_log_pid_match (log_sliding_window.cpp:3913) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A463D-0-0] [lt=18][errcode=-4018] get_prev_log_info_ failed(tmp_ret=-4018, log_id=18160, palf_id=1002, self="192.168.159.137:2882", log_id=18160, lsn={lsn:2210320}, prev_lsn={lsn:2210198}, prev_log_pid=1) [2025-01-08 00:51:23.234750] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] no need to update min/max version(new_version={val:1736326282603912001, v:0}, min_version={val:1736326282603912001, v:0}, max_version={val:1736326282603912001, v:0}) [2025-01-08 00:51:23.235303] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58520][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:23.235673] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A92A6-0-0] [lt=38] init point thread id with(&point=0x5572d50e4f40, idx_=1176, point=[thread id=58520, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58520) [2025-01-08 00:51:23.236507] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB600-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:23.236570] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB600-0-0] [lt=49] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326282603912004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326282603912004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:23.237098] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84330646}], scn_array=[cnt:1, {val:1736326283039287001, v:0}], curr_size=122, accum_size=708, time_cost=38318) [2025-01-08 00:51:23.237262] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=61] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=28, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:23.237313] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=192868, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:23.237499] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F8A-00062B2D1E9A92A5-0-0] [lt=20][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5398, fly_ts=126027, send_timestamp=1736326283111470, peer="192.168.159.139:45062", sz=300) [2025-01-08 00:51:23.237938] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A463B-0-0] [lt=28][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18158, log_task={header:{begin_lsn:{lsn:2210076}, end_lsn:{lsn:2210198}, log_id:18158, min_scn:{val:1736326282603912002, v:0}, max_scn:{val:1736326282603912002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2209954}, prev_proposal_id:1, committed_end_lsn:{lsn:2210076}, data_checksum:1715074492, accum_checksum:2612049775, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326282996870, freeze_ts:1736326282996870, submit_ts:1736326282996878, flushed_ts:1736326282999402, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:2524}, fs_cb_begin_ts=1736326283237929, log_life_time=241059) [2025-01-08 00:51:23.239576] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:23.240975] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D1A8BB600-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326283236819000, start_id=1736326283236819000, end_id=1736326283236819001) [2025-01-08 00:51:23.246109] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E57-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326283236819000, start_id=1736326283236819001, end_id=1736326283236819002) [2025-01-08 00:51:23.249553] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:2, {lsn:2210198}, {lsn:2210320}], scn_array=[cnt:2, {val:1736326282603912003, v:0}, {val:1736326282603912004, v:0}], curr_size=244, accum_size=366, time_cost=11438) [2025-01-08 00:51:23.250947] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43284565}], scn_array=[cnt:1, {val:1736326283180565000, v:0}], curr_size=589, accum_size=2606, time_cost=69152) [2025-01-08 00:51:23.250959] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2197799}], scn_array=[cnt:1, {val:1736326282603912005, v:0}], curr_size=122, accum_size=366, time_cost=16771) [2025-01-08 00:51:23.258799] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=27] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8683, offset=2464825}, end_cursor:ObLogCursor{file_id=1, log_id=8684, offset=2465122}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894aa0050, len:967, log_data_len:297, seq:8683, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:23.259304] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E57-0-0] [lt=45] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:23.259477] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E57-0-0] [lt=102] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326282603912002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326282603912002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:23.260878] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=25] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:23.265770] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:23.271090] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.273148] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=37] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8684, offset=2465122}, end_cursor:ObLogCursor{file_id=1, log_id=8685, offset=2465419}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894aa4050, len:670, log_data_len:297, seq:8684, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:23.277075] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=83] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8685, offset=2465419}, end_cursor:ObLogCursor{file_id=1, log_id=8686, offset=2465716}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894aa8050, len:373, log_data_len:297, seq:8685, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:23.277653] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] ====== check clog disk timer task ====== [2025-01-08 00:51:23.277740] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=79] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:23.297396] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:23.302766] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE889-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326283301126000, start_id=1736326283301126000, end_id=1736326283301126001) [2025-01-08 00:51:23.314417] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D41-0-0] [lt=7] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:23.316968] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=12478464) [2025-01-08 00:51:23.317125] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=161] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:23.318939] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2197921}], scn_array=[cnt:1, {val:1736326282603912006, v:0}], curr_size=122, accum_size=488, time_cost=12166) [2025-01-08 00:51:23.319580] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.319762] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.334242] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:23.334790] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:23.340882] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58503][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=152] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:23.349560] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:23.364318] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1736326283364314, bytes: 875712942, bw: 0.152675 MB/s, add_ts: 1003423, add_bytes: 160639 [2025-01-08 00:51:23.365263] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A5F-0-0] [lt=133] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:23.373786] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB9-0-0] [lt=18] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:23.373821] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CB9-0-0] [lt=32] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:23.374807] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.374952] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2025-01-08 00:51:23.383407] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:23.454842] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:23.456513] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=26] gc stale ls task succ [2025-01-08 00:51:23.456675] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] no need to update min/max version(new_version={val:1736326283039287000, v:0}, min_version={val:1736326283039287000, v:0}, max_version={val:1736326283039287000, v:0}) [2025-01-08 00:51:23.456679] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=194] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1289696, replace_num=15728) [2025-01-08 00:51:23.456708] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:23.457094] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=124] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:23.458309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.459468] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=180] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.471270] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23] ====== tenant freeze timer task ====== [2025-01-08 00:51:23.473432] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:23.473578] WDIAG read_multi_row (ob_table_access_helper.h:238) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:23.475863] INFO delete_row (ob_table_access_helper.h:339) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1001, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326283473595) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326283473595) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:23.476566] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.498837] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] ====== tenant freeze timer task ====== [2025-01-08 00:51:23.498893] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=46] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71333860, tenant_id=1001) [2025-01-08 00:51:23.498969] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=67] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:23.503716] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=79] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:23.505852] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=99] clear dtl interm result cost(us)(clear_cost=9733, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:23.510367] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58519][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C0E-0-0] [lt=123] init point thread id with(&point=0x5572d50e4f00, idx_=1175, point=[thread id=58519, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58519) [2025-01-08 00:51:23.519505] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.522152] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AA-0-0] [lt=52] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:23.522241] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AA-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326283307390000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326283307390000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:23.542430] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.544878] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=76] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.577011] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.586027] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:194) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=56] get wrs ts(ls_id={id:1}, delta=288797, log_service_ts=1736326283297023, min_tx_service_ts=4611686018427387, timestamp={val:1736326283297023001, v:0}) [2025-01-08 00:51:23.586136] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=101] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=7, Other_Error_Cnt=0, Submit_Succ_Cnt=80, last_scn="{val:1736326277123056004, v:0}", last_lsn={lsn:18679324}, last_gts={val:1736326283297023000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-01-08 00:51:23.656909] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:23.657089] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1736326283558449000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1736326283656898}, version_delta=98637) [2025-01-08 00:51:23.657797] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=56] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:23.671816] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BA7-0-0] [lt=10] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:23.672671] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43290893}, readable_log_tail:{lsn:43290893}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23813875, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:23.677149] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:23.677298] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=107] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:8, bw:34377, [delay/us]:prepare:7, schedule:71, submit:58725, rt:5800, total:64605) [2025-01-08 00:51:23.677318] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:23.677377] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:9, avg_bw:34383, [delay/us]: prepare:7, schedule:71, submit:58725, device:5800, total:64605) [2025-01-08 00:51:23.677391] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:23.677429] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:16, bw:76409, [delay/us]:prepare:13, schedule:3527, submit:278, rt:37155, total:40975) [2025-01-08 00:51:23.677439] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:23.677475] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:16, avg_bw:76415, [delay/us]: prepare:13, schedule:3527, submit:278, device:37155, total:40975) [2025-01-08 00:51:23.677485] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:23.677507] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:7, bw:26746, [delay/us]:prepare:6, schedule:113, submit:1, rt:15248, total:15370) [2025-01-08 00:51:23.677529] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:2, hold_mem:16384 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4220, iops:31, bw:129909, [delay/us]:prepare:5, schedule:490, submit:423, rt:53390, total:54309) [2025-01-08 00:51:23.677538] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:11462, [delay/us]:prepare:2, schedule:72, submit:878, rt:704, total:1658) [2025-01-08 00:51:23.677545] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=6, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49998, free_result_cnt=49998, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=8, iops=8, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:23.677573] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:7, avg_bw:26746, [delay/us]: prepare:6, schedule:113, submit:1, device:15248, total:15370) [2025-01-08 00:51:23.677588] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:11462, [delay/us]: prepare:2, schedule:72, submit:878, device:704, total:1658) [2025-01-08 00:51:23.677596] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4220, avg_iops:31, avg_bw:129911, [delay/us]: prepare:5, schedule:490, submit:423, device:53390, total:54309) [2025-01-08 00:51:23.677605] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:23.677625] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3820, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:216256, total:216267) [2025-01-08 00:51:23.677648] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:23.677818] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=164] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3820, [delay/us]: prepare:0, schedule:0, submit:600000000, device:216256, total:216267) [2025-01-08 00:51:23.677849] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:23.677964] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:23.677977] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:23.677985] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:23.677994] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=209kB/s, net_out=184kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:23.678356] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.679382] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BECC-0-0] [lt=9] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:23.694700] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=43] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:23.694726] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:23.694802] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84986-0-0] [lt=9] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:23.694885] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84986-0-0] [lt=78] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:23.720901] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] BlockGCTimerTask success(ret=0, cost_time_us=26, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:23.753580] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=38] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:23.754813] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=58] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:23.755087] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=53] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326274317657000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:23.758385] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E58-0-0] [lt=91] InfoPoolResizeTask(cost_ts=8) [2025-01-08 00:51:23.778759] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.780376] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=72] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:23.781734] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=42] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:23.781837] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326275608151000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:23.781865] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326275608151000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326275608151000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:23.785160] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43290893}], scn_array=[cnt:1, {val:1736326283581997001, v:0}], curr_size=122, accum_size=8467, time_cost=13526) [2025-01-08 00:51:23.789393] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=57] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=17231634, this={dio_aligned_buf:{buf_write_offset:348, buf_padding_size:3748, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:70695, truncate_used_ts:17434}, log_block_size:67108864, io_fd:{first_id:-1, second_id:152, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=8192, aligned_block_offset=17227776, buf_len=586, total_write_size=36135, total_write_size_after_dio=425984, ob_pwrite_used_ts=1601039, count=95, each_pwrite_cost=16853) [2025-01-08 00:51:23.793862] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=169] ====== check clog disk timer task ====== [2025-01-08 00:51:23.793905] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=38] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:23.795766] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:23.798198] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:202) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=70] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2025-01-08 00:51:23.798287] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=85] finish to resize log external storage handler(current_ts=1736326283798283, tenant_max_cpu=4, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:23.798318] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=29] snapshot_traffic_info once [2025-01-08 00:51:23.798427] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=138] get number(ret=0, service_type_=0, range=1, base_id=1736326283797570000, start_id=1736326283797570000, end_id=1736326283797570001) [2025-01-08 00:51:23.798932] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46205][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=27] submit log success(service_type_=0, last_id=1736326300087570000, limited_id=1736326300087570000) [2025-01-08 00:51:23.798948] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46205][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15] submit log with lock success(service_type_=0, last_id=1736326300087570000, limited_id=1736326300087570000) [2025-01-08 00:51:23.799320] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=14] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:320K/75:1037297 disk:320K/75:1037297 t1m2:44K/10:70944 t1m2g0:44K/10:70944 t1002m2:192K/47:364184 t1002m2g0:192K/47:364184 t1001m2:84K/18:3329714 t1001m2g0:84K/18:3329714 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:23.799330] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1003}, {id:1002}]) [2025-01-08 00:51:23.799458] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:23.799490] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:51:23.799548] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=9] gc stale ls task succ [2025-01-08 00:51:23.804314] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:23.804357] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:23.806531] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:263) [50687][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D2008458E-0-0] [lt=29] ObArchiveSender is running(thread_index=4) [2025-01-08 00:51:23.807718] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE88C-0-0] [lt=54] is logging [2025-01-08 00:51:23.807738] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE88C-0-0] [lt=16] submit log with lock(ret=-4023, service_type_=0, last_id=1736326300087570000, limited_id=1736326300087570000) [2025-01-08 00:51:23.808566] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=48] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:23.808818] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38E9-0-0] [lt=36][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18062, log_task={header:{begin_lsn:{lsn:2198409}, end_lsn:{lsn:2198531}, log_id:18062, min_scn:{val:1736326283301126003, v:0}, max_scn:{val:1736326283301126003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2198287}, prev_proposal_id:1, committed_end_lsn:{lsn:2198409}, data_checksum:940009244, accum_checksum:765024815, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326283706826, freeze_ts:1736326283706826, submit_ts:1736326283706836, flushed_ts:1736326283708670, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:1834}, fs_cb_begin_ts=1736326283808804, log_life_time=101978) [2025-01-08 00:51:23.811069] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918542E-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:23.811358] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:23.811378] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-01-08 00:51:23.811388] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:23.811402] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:23.811417] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193787716612, flush_io_finished_ret_:0, flush_io_finished_round_:1942, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:23.811456] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:23.811468] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] temporary file module use no blocks [2025-01-08 00:51:23.811473] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:23.811482] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:23.811491] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193787716624, flush_io_finished_ret_:0, flush_io_finished_round_:3770, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:23.814047] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE88A-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1736326283811422000, start_id=1736326283811422000, end_id=1736326283811422001) [2025-01-08 00:51:23.817151] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18686740}], scn_array=[cnt:1, {val:1736326283797570001, v:0}], curr_size=128, accum_size=1104, time_cost=18069) [2025-01-08 00:51:23.817338] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] update id meta(ret=0, service_type=0, limited_id=1736326300087570000, latest_scn={val:1736326283797570001, v:0}, *this={tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326280628634000, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326300087570000, latest_log_ts:{val:1736326283797570001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:23.817397] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=58] handle submit callback(service_type_=0, success=true, used_ts=18466, log_ts={val:1736326283797570001, v:0}, limited_id=1736326300087570000) [2025-01-08 00:51:23.817410] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=12] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:23.817420] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=9] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:23.819999] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=41] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:23.821823] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=104] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:23.821968] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=52] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326275719333000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:23.822026] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=55] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326275719333000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326275719333000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:23.823362] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:23.823457] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:23.826947] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F522-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1736326283826027000, start_id=1736326283826027000, end_id=1736326283826027001) [2025-01-08 00:51:23.827918] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870C4-0-0] [lt=8][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:23.828008] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:23.828024] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:23.830055] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=56] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:23.835161] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3C8-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:23.835815] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=94] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:23.836008] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=139] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:23.841071] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:23.849740] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=37] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:23.853274] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:23.853905] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA2-0-0] [lt=54][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:23.854929] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58509][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=39] worker thread exit(tenant_->id()=1001, workers_.get_size()=3) [2025-01-08 00:51:23.855564] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1736326283855560, bytes: 0, bw: 0.000000 MB/s, add_ts: 1013627, add_bytes: 0 [2025-01-08 00:51:23.857628] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58516][TimerWK5][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:23.859045] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA2-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:23.860709] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D062-0-0] [lt=79][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:23.862770] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:23.878310] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2953, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:23.878418] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=98] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=760, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:23.878457] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=92] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:23.878924] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.879495] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A92BC-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:23.879517] INFO [LIB] stat (utility.h:1210) [58512][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A92BC-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11934, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:23.897394] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=53] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.897668] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.897777] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.897809] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:23.901917] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38EB-0-0] [lt=184] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:23.903242] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58490][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8689-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:23.903420] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=159, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:23.903442] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=8, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:23.903558] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=91] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4756, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:23.903580] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=733, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:23.903619] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=32, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:23.905790] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58468][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8C-0-0] [lt=32] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326284012828, keepalive_ts:-1}) [2025-01-08 00:51:23.905820] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58468][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8C-0-0] [lt=28] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326284012828, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:23.915494] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=42] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:23.916572] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=54] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:23.916884] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C664-0-0] [lt=33][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193788830661, cache_obj->added_lc()=false, cache_obj->get_object_id()=13632, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:23.917637] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5862, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:23.917696] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1020, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:23.917750] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=34, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:23.971593] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=32] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=9711, avg_group_log_size=571) [2025-01-08 00:51:23.971700] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=100] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:23.974402] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58505][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D3F-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9711) [2025-01-08 00:51:23.975274] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=99] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=7100, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:23.975316] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9711) [2025-01-08 00:51:23.975348] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=250, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:23.975618] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=21, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:23.975639] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=46334, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:23.975710] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=69] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=1304, avg_log_gen_to_submit_time=1392, avg_log_submit_to_flush_time=8028, avg_log_submit_to_slide_time=12257) [2025-01-08 00:51:23.975851] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9711, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:23.975872] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=81] [PALF STAT FLUSH CB COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=93, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:23.976849] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=92, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:23.976891] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1226, avg_group_log_size=122) [2025-01-08 00:51:23.976909] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1226) [2025-01-08 00:51:23.976921] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:23.976987] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=39, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:23.977006] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:23.977018] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:23.977028] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:23.977953] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:23.978017] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:23.978068] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=92] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1226) [2025-01-08 00:51:23.978099] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1226) [2025-01-08 00:51:23.978985] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=96] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:19528, max_lsn:18686990, max_scn:{val:1736326283797570002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:18686990}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:18686868}, last_submit_end_lsn:{lsn:18686990}, last_submit_log_id:19528, last_submit_log_pid:1, max_flushed_lsn:{lsn:18686868}, max_flushed_end_lsn:{lsn:18686990}, max_flushed_log_pid:1, committed_end_lsn:{lsn:18686868}, last_slide_log_id:19527, last_slide_scn:{val:1736326283797570001, v:0}, last_slide_lsn:{lsn:18686740}, last_slide_end_lsn:{lsn:18686868}, last_slide_log_pid:1, last_slide_log_accum_checksum:1289125895, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8fc454530}, [StateMgr]={this:0x2ad8fc458830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:19528, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324399554770, palf_role_change_cb:0x2ad8fc45f2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8fc455c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324399758968, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/meta", dir_fd:175, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8fc45aa38, flashback_version:0}, log_storage:{log_tail:{lsn:18686990}, readable_log_tail:{lsn:18686990}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48417778, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324399554279, this:0x2ad8fc458f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324399555093, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8fc4589b0}) [2025-01-08 00:51:23.979762] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=611] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326283978095, last_flushed_end_lsn:{lsn:18686990}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326283825889, last_flushed_end_lsn:{lsn:18686868}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326283811003, last_flushed_end_lsn:{lsn:18686868}}]) [2025-01-08 00:51:23.979792] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=19528, log_task={header:{begin_lsn:{lsn:18686868}, end_lsn:{lsn:18686990}, log_id:19528, min_scn:{val:1736326283797570002, v:0}, max_scn:{val:1736326283797570002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18686868}, data_checksum:667309872, accum_checksum:485064234, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1736326283976805, freeze_ts:1736326283976809, submit_ts:1736326283976870, flushed_ts:1736326283978094, gen_to_freeze cost time:4, gen_to_submit cost time:65, submit_to_flush cost time:1224}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326283978095, last_flushed_end_lsn:{lsn:18686990}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326283825889, last_flushed_end_lsn:{lsn:18686868}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326283811003, last_flushed_end_lsn:{lsn:18686868}}]) [2025-01-08 00:51:23.980077] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=148] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1002, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18167, max_lsn:2211296, max_scn:{val:1736326283826027000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2211296}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2211174}, last_submit_end_lsn:{lsn:2211296}, last_submit_log_id:18167, last_submit_log_pid:1, max_flushed_lsn:{lsn:2211174}, max_flushed_end_lsn:{lsn:2211296}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2211296}, last_slide_log_id:18167, last_slide_scn:{val:1736326283826027000, v:0}, last_slide_lsn:{lsn:2211174}, last_slide_end_lsn:{lsn:2211296}, last_slide_log_pid:1, last_slide_log_accum_checksum:1248559052, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad92435e530}, [StateMgr]={this:0x2ad924362830, self:"192.168.159.137:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.138:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18168, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad9243692f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326280764668, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326280764668, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad92435fc70}, [ModeMgr]={palf_id:1002, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436803442001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/meta", dir_fd:184, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924364a38, flashback_version:0}, log_storage:{log_tail:{lsn:2211296}, readable_log_tail:{lsn:2211296}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64893472, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:-1, this:0x2ad924362f70}, [Reconfirm]={palf_id:1002, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9243629b0}) [2025-01-08 00:51:23.980370] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=288][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:23.980385] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:23.980394] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:23.980403] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] current log_task status(palf_id=1002, self="192.168.159.137:2882", log_id=18168, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:51:23.980526] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:23.981127] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58522][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:23.981291] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58522][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39D8-0-0] [lt=18] init point thread id with(&point=0x5572d50e4fc0, idx_=1178, point=[thread id=58522, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58522) [2025-01-08 00:51:23.981628] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC868A-0-0] [lt=126] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:23.981642] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC868A-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7745, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:23.981651] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC868A-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=113, avg_log_submit_to_flush_time=5781, avg_log_submit_to_slide_time=7299) [2025-01-08 00:51:23.981764] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1226, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:23.982042] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread created(tenant_->id()=1001, group_id_=1, token=3) [2025-01-08 00:51:23.988848] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:23.990626] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=42] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:23.992271] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=166] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326283784079000, v:0}, new_max_version={val:1736326283784079000, v:0}, last_min_version={val:1736326283574361000, v:0}, last_max_version={val:1736326283574361000, v:0}, total_time_us=19093, delta_us=208094) [2025-01-08 00:51:23.992406] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=130] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326283784079000, v:0}, min_version={val:1736326283784079000, v:0}, max_version={val:1736326283784079000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:23.993939] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread created(tenant_->id()=1001, group_id_=1, token=4) [2025-01-08 00:51:23.993944] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58523][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:23.997264] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A464D-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:23.997377] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A464D-0-0] [lt=101] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:23.999230] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=136] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3656, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:23.999325] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=91] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:23.999371] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=548, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:23.999432] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=75, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:24.001784] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D859-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=23, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:24.001879] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D859-0-0] [lt=94] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=40646, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:24.001894] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D859-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=307, avg_log_submit_to_flush_time=5020, avg_log_submit_to_slide_time=40339) [2025-01-08 00:51:24.001984] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:24.002035] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18076, miss_cnt:0, cache_read_size:2211418, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:24.003542] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38ED-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:24.003605] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC38ED-0-0] [lt=52] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:24.004715] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:24.014582] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:24.016611] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58468][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=15] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-01-08 00:51:24.057777] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5252743304, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:24.057975] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=67] recycle one thread(this=0x2ad8885abcf0, total=4, remain=3) [2025-01-08 00:51:24.058004] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=28] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5252743304, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:24.058023] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15058026368, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:24.058253] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=128] recycle one thread(this=0x2ad8881fc3b0, total=5, remain=4) [2025-01-08 00:51:24.058270] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15058026368, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:24.058318] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58516][TimerWK5][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:24.063241] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:24.080296] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5130, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:24.080336] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=741, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:24.080363] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:24.080706] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.109993] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3662) [2025-01-08 00:51:24.110034] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:24.112784] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3662) [2025-01-08 00:51:24.112839] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3662) [2025-01-08 00:51:24.114425] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58519][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C14-0-0] [lt=52] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=2513, avg_log_gen_to_submit_time=2619, avg_log_submit_to_flush_time=6081, avg_log_submit_to_slide_time=7712) [2025-01-08 00:51:24.114584] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=79] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3662, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:24.171787] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=67, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:24.171834] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9582) [2025-01-08 00:51:24.173308] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=41] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:24.181051] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.206719] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D85E-0-0] [lt=16] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=5133, avg_log_submit_to_slide_time=45658) [2025-01-08 00:51:24.206879] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:24.206915] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17977, miss_cnt:0, cache_read_size:2199141, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:24.243253] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:24.243431] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58490][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=39] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:24.256286] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A79-0-0] [lt=16] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326284256269}) [2025-01-08 00:51:24.259455] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15126434977, tenant_id:500}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:24.259560] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=39] recycle one thread(this=0x5572d527f900, total=7, remain=6) [2025-01-08 00:51:24.259575] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15126434977, tenant_id:500}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:24.259588] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7857160645, tenant_id:1002}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:24.259649] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] recycle one thread(this=0x2ad8c0eb0e70, total=7, remain=6) [2025-01-08 00:51:24.259660] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7857160645, tenant_id:1002}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:24.263594] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597A-0-0] [lt=17] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:24.263871] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:24.264561] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58515][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:24.267507] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=77] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:24.273538] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB601-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:24.273589] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB601-0-0] [lt=33] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326283807366003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326283807366003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:24.277327] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=56] ls blacklist refresh finish(cost_time=22675) [2025-01-08 00:51:24.283239] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1A8BB601-0-0] [lt=46] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.840151977539062500e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=392, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.431660989222258422e+02) [2025-01-08 00:51:24.283837] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597A-0-0] [lt=41] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326284283806, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326284283806, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326284283806, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:24.284073] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597A-0-0] [lt=233] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326284256269}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326284283806, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326284283806, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326284283806, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:24.284128] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.292642] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8686, offset=2465716}, end_cursor:ObLogCursor{file_id=1, log_id=8687, offset=2466021}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894aac050, len:4172, log_data_len:305, seq:8686, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:24.294636] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E58-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:24.294686] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E58-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326283797570005, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326283797570005, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:24.295635] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50625][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB601-0-0] [lt=51] get number(ret=0, service_type_=0, range=1, base_id=1736326284291649000, start_id=1736326284291649000, end_id=1736326284291649001) [2025-01-08 00:51:24.299425] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7] ts response task statistics(alloc_count=60194, free_count=30096) [2025-01-08 00:51:24.299476] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:24.300021] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=423, get_gts_with_stc_cnt=3, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=3, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:24.300806] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4653-0-0] [lt=60][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18170, log_task={header:{begin_lsn:{lsn:2211540}, end_lsn:{lsn:2211662}, log_id:18170, min_scn:{val:1736326283826027003, v:0}, max_scn:{val:1736326283826027003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2211418}, prev_proposal_id:1, committed_end_lsn:{lsn:2211540}, data_checksum:1715074492, accum_checksum:3759023716, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326284197622, freeze_ts:1736326284197622, submit_ts:1736326284197631, flushed_ts:1736326284199227, gen_to_freeze cost time:0, gen_to_submit cost time:9, submit_to_flush cost time:1596}, fs_cb_begin_ts=1736326284300794, log_life_time=103172) [2025-01-08 00:51:24.310528] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=35] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:24.311290] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=101, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:24.311374] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=72] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=3662, avg_group_log_size=332) [2025-01-08 00:51:24.316353] INFO [LIB] stat (utility.h:1210) [58478][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C16-0-0] [lt=42] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:24.316391] INFO [LIB] stat (utility.h:1210) [58478][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C16-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7248, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:24.316570] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D860-0-0] [lt=48] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=9, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:24.316590] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D860-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=42505, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:24.317024] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=51] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-01-08 00:51:24.317488] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8687, offset=2466021}, end_cursor:ObLogCursor{file_id=1, log_id=8688, offset=2466318}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ab0050, len:3867, log_data_len:297, seq:8687, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:24.318984] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=43] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:24.319121] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=130] temporary file module use no blocks [2025-01-08 00:51:24.319133] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:24.319147] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:24.319160] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193788232238, flush_io_finished_ret_:0, flush_io_finished_round_:1947, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:24.319282] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.319873] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E58-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:24.319938] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E58-0-0] [lt=43] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326283797570004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326283797570004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:24.320248] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.324974] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:24.326699] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:24.326868] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:24.326991] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:24.333231] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8688, offset=2466318}, end_cursor:ObLogCursor{file_id=1, log_id=8689, offset=2466615}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ab4050, len:3570, log_data_len:297, seq:8688, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:24.337140] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=108] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:24.337346] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:24.345403] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58506][T1002_TimerWK5][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:24.349291] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=49] Slow write(duration=11663, write_len=4096) [2025-01-08 00:51:24.349345] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=51] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8689, offset=2466615}, end_cursor:ObLogCursor{file_id=1, log_id=8690, offset=2466912}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ab8050, len:3273, log_data_len:297, seq:8689, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:24.350776] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58503][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:24.359455] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:112) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690EB1-0-0] [lt=223] adaptive update(Tenant ID=1, Config Specify Resource Limit(MB)=1023, TxShare Current Memory Limit(MB)=894, Holding Memory(MB)=140, Tenant Remain Memory(MB)=1257, Usable Remain Memory(MB)=754, Last Update Limit Timestamp=1736326284359372, Is Updated=true) [2025-01-08 00:51:24.361999] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:51:24.362003] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58524][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:24.366013] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=47] PNIO [ratelimit] time: 1736326284366010, bytes: 875972829, bw: 0.247428 MB/s, add_ts: 1001696, add_bytes: 259887 [2025-01-08 00:51:24.371746] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=44] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:24.371855] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=102] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:24.371872] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=16] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:24.371894] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:24.375086] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:24.376057] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=967] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:24.376178] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBA-0-0] [lt=14] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:24.376272] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBA-0-0] [lt=90] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:24.376587] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31] LogLoopThread round_cost_time(us)(round_cost_time=16) [2025-01-08 00:51:24.380172] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=43] LogLoopThread round_cost_time(us)(round_cost_time=55) [2025-01-08 00:51:24.383812] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=97] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326283797570003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=586233, in_cluster_service=true, cluster_version={val:1736326283797570001, v:0}, min_cluster_version={val:1736326283797570001, v:0}, max_cluster_version={val:1736326283797570001, v:0}, get_cluster_version_err=0, cluster_version_delta=586233, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18378, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=512301330, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:24.384694] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.387568] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:24.387572] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58525][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:24.431782] INFO dump_info (ob_timer_service.cpp:652) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=9] dump info [summary](this=0x2ad8f90b0af0, this={this:0x2ad8f90b0af0, tenant_id_:1001, priority_task_queue_.size():43, running_task_set_.size():0, uncanceled_task_set_.size():0, worker_thread_pool_.get_queue_num():-4}) [2025-01-08 00:51:24.431967] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=184] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604578, timer:0x2ad8dd4ddf40, task:0x2ad8fb605038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1736326284431643, delay:20000}) [2025-01-08 00:51:24.431984] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=18] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16043f8, timer:0x2ad8dd4ddf40, task:0x2ad8fb605010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1736326284441240, delay:20000}) [2025-01-08 00:51:24.431992] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16046f8, timer:0x2ad8b89d3f40, task:0x2ad8f97e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1736326284450583, delay:30000}) [2025-01-08 00:51:24.431999] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604db8, timer:0x2ad8f959d040, task:0x2ad8f959d088, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService16ObCheckpointTaskE", scheduled_time:1736326284812501, delay:5000000}) [2025-01-08 00:51:24.432005] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605e38, timer:0x2ad8dd4c9b70, task:0x2ad8f95c8820, task_type:"N9oceanbase5table21ObTableApiSessPoolMgr29ObTableApiSessEliminationTaskE", scheduled_time:1736326284823689, delay:5000000}) [2025-01-08 00:51:24.432011] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16041b8, timer:0x2ad8b8821f40, task:0x2ad8f959ab60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1736326284835765, delay:1000000}) [2025-01-08 00:51:24.432018] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605b38, timer:0x2ad8f95bec70, task:0x2ad8f95bed00, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1736326284835786, delay:1000000}) [2025-01-08 00:51:24.432024] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605a78, timer:0x2ad8f95bec70, task:0x2ad8f95bece8, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1736326284836243, delay:1000000}) [2025-01-08 00:51:24.432030] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604c38, timer:0x2ad8dbc81f40, task:0x2ad8ff526c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1736326284837166, delay:500000}) [2025-01-08 00:51:24.432036] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604b78, timer:0x2ad8dbc81f40, task:0x2ad8ff526bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1736326284837389, delay:500000}) [2025-01-08 00:51:24.432042] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605778, timer:0x2ad8f95bec70, task:0x2ad8f95bec88, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1736326284838081, delay:1000000}) [2025-01-08 00:51:24.432049] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16059b8, timer:0x2ad8f95bec70, task:0x2ad8f95becd0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1736326284838120, delay:1000000}) [2025-01-08 00:51:24.432055] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16047b8, timer:0x2ad8dd4edf40, task:0x2ad8f97e93c0, task_type:"N9oceanbase4palf10LogUpdaterE", scheduled_time:1736326284840695, delay:500000}) [2025-01-08 00:51:24.432061] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604338, timer:0x2ad8dd5e5330, task:0x2ad8f95bf360, task_type:"N9oceanbase3sql16ObUDRRefreshTaskE", scheduled_time:1736326284854737, delay:5000000}) [2025-01-08 00:51:24.432067] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16049f8, timer:0x2ad8dd4e5f40, task:0x2ad8fb7e82a0, task_type:"N9oceanbase8observer28ObTenantLSMetaTableCheckTaskE", scheduled_time:1736326284857592, delay:0}) [2025-01-08 00:51:24.432073] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604ab8, timer:0x2ad8dd4c9b70, task:0x2ad8f95c8558, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1736326284871264, delay:0}) [2025-01-08 00:51:24.432080] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604f38, timer:0x2ad8f959d070, task:0x2ad8f959d0a8, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService24ObCheckClogDiskUsageTaskE", scheduled_time:1736326285175293, delay:2000000}) [2025-01-08 00:51:24.432086] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604ff8, timer:0x2ad8f959d188, task:0x2ad8f959d1a0, task_type:"N9oceanbase7storage10checkpoint17ObTabletGCService18ObTabletChangeTaskE", scheduled_time:1736326285658093, delay:5000000}) [2025-01-08 00:51:24.432092] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16055f8, timer:0x2ad88564dc70, task:0x2ad8f95bc038, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1736326286755228, delay:3000000}) [2025-01-08 00:51:24.432104] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604e78, timer:0x2ad8f959d058, task:0x2ad8f959d098, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService20ObTraversalFlushTaskE", scheduled_time:1736326287423426, delay:5000000}) [2025-01-08 00:51:24.432112] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16040f8, timer:0x2ad8dd5dd110, task:0x2ad8f9559b68, task_type:"N9oceanbase3sql26ObPlanCacheEliminationTaskE", scheduled_time:1736326287457471, delay:5000000}) [2025-01-08 00:51:24.432119] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16050b8, timer:0x2ad8f959d1b8, task:0x2ad8f959d1d0, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1736326287837062, delay:5000000}) [2025-01-08 00:51:24.432145] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=24] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604878, timer:0x2ad8dd59b110, task:0x2ad8f94044d0, task_type:"N9oceanbase10logservice17ReplayProcessStatE", scheduled_time:1736326288762451, delay:10000000}) [2025-01-08 00:51:24.432155] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605838, timer:0x2ad8f95bec70, task:0x2ad8f95beca0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1736326288762544, delay:10000000}) [2025-01-08 00:51:24.432163] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604cf8, timer:0x2ad8dd4c9b70, task:0x2ad8f955be50, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1736326288769318, delay:10000000}) [2025-01-08 00:51:24.432170] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16044b8, timer:0x2ad8dd4ddf40, task:0x2ad8fb605028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1736326289108249, delay:10000000}) [2025-01-08 00:51:24.432177] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604038, timer:0x2ad8db9d3f40, task:0x2ad8ff004038, task_type:"N9oceanbase7storage21ObTenantTabletStatMgr17TabletStatUpdaterE", scheduled_time:1736326289110373, delay:5000000}) [2025-01-08 00:51:24.432183] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605538, timer:0x2ad8db993f40, task:0x2ad8f95b3a80, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1736326290441247, delay:20000000}) [2025-01-08 00:51:24.432241] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16056b8, timer:0x2ad88564dc70, task:0x2ad8f95bc050, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1736326290736837, delay:10000000}) [2025-01-08 00:51:24.432252] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605238, timer:0x2ad8dd4e3f40, task:0x2ad8f95b3a50, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr13SSTableGCTaskE", scheduled_time:1736326292671848, delay:30000000}) [2025-01-08 00:51:24.432266] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=14] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604638, timer:0x2ad8dd579110, task:0x2ad8fb681750, task_type:"N9oceanbase12blocksstable21ObSharedMacroBlockMgr26ObBlockDefragmentationTaskE", scheduled_time:1736326313006882, delay:30000000}) [2025-01-08 00:51:24.432273] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604938, timer:0x2ad8dd59b6d0, task:0x2ad8fb77a518, task_type:"N9oceanbase7storage29ObTenantCheckpointSlogHandler21ObWriteCheckpointTaskE", scheduled_time:1736326313127107, delay:60000000}) [2025-01-08 00:51:24.432281] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16058f8, timer:0x2ad8f95bec70, task:0x2ad8f95becb8, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1736326313722508, delay:30000000}) [2025-01-08 00:51:24.432304] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=21] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16052f8, timer:0x2ad8dd4eff40, task:0x2ad8f95b3a60, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr18InfoPoolResizeTaskE", scheduled_time:1736326313758411, delay:30000000}) [2025-01-08 00:51:24.432314] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605fb8, timer:0x2ad8dd4c9b70, task:0x2ad8ff5f3ee0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1736326313981607, delay:60000000}) [2025-01-08 00:51:24.432321] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605d78, timer:0x2ad8dd4c9b70, task:0x2ad8f9604030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1736326316367552, delay:0}) [2025-01-08 00:51:24.432328] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1604278, timer:0x2ad8dd4e1f40, task:0x2ad8f959d678, task_type:"N9oceanbase7storage25ObTenantCompactionMemPool17MemPoolShrinkTaskE", scheduled_time:1736326372761090, delay:120000000}) [2025-01-08 00:51:24.432334] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605178, timer:0x2ad8dd597a80, task:0x2ad8f95b3a30, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr13MergeLoopTaskE", scheduled_time:1736326374564485, delay:120000000}) [2025-01-08 00:51:24.432340] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605478, timer:0x2ad8db993f40, task:0x2ad8f95b3a40, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1736326390441141, delay:120000000}) [2025-01-08 00:51:24.432346] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d16053b8, timer:0x2ad8dd4eff40, task:0x2ad8f95b3a70, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr24TabletUpdaterRefreshTaskE", scheduled_time:1736326433403127, delay:300000000}) [2025-01-08 00:51:24.432423] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=76] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605cb8, timer:0x2ad8dd4c9b70, task:0x2ad8f95c7be0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1736327033976116, delay:900000000}) [2025-01-08 00:51:24.432434] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1606078, timer:0x2ad8db8a7f40, task:0x2ad8fb7e82b8, task_type:"N9oceanbase8observer32ObTenantTabletMetaTableCheckTaskE", scheduled_time:1736327933375199, delay:0}) [2025-01-08 00:51:24.432442] INFO dump_info (ob_timer_service.cpp:655) [50562][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2ad8f90b0af0, token={this:0x2ad8d1605bf8, timer:0x2ad8dd4c9b70, task:0x2ad8f95c7bc8, task_type:"N9oceanbase6common28ObOptStatMonitorFlushAllTaskE", scheduled_time:1736410733570704, delay:86400000000}) [2025-01-08 00:51:24.456689] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=27] gc stale ls task succ [2025-01-08 00:51:24.458118] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] get_max_applied_scn(scn={val:1736326284359372000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84335016}, palf_committed_end_scn_:{val:1736326284359372000, v:0}, last_check_scn_:{val:1736326284359372000, v:0}, max_applied_cb_scn_:{val:1736326284359372000, v:0}}) [2025-01-08 00:51:24.458238] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84335016}, readable_log_tail:{lsn:84335016}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49874520, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:24.459281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.459563] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.464115] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:24.465128] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58515][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:24.476651] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:24.476742] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=88] try gc retain ctx [2025-01-08 00:51:24.476753] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-01-08 00:51:24.476792] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] get_max_applied_scn(scn={val:1736326284371140000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43299880}, palf_committed_end_scn_:{val:1736326284383131000, v:0}, last_check_scn_:{val:1736326284383131000, v:0}, max_applied_cb_scn_:{val:1736326284371140000, v:0}}) [2025-01-08 00:51:24.476822] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] finish update min start scn(max_decided_scn={val:1736326284371140000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326274427888001, v:0}, keep_alive_scn_:{val:1736326279403283001, v:0}, update_ts_:1736326284475781}) [2025-01-08 00:51:24.477587] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299284252000, latest_log_ts:{val:1736326282955407001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:24.485127] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.510258] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=60] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:24.511133] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:24.511227] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=91] try gc retain ctx [2025-01-08 00:51:24.511239] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] try to retry start_working [2025-01-08 00:51:24.511268] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326284359372000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326274393594001, v:0}, keep_alive_scn_:{val:1736326279315890001, v:0}, update_ts_:1736326284511029}) [2025-01-08 00:51:24.511963] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=76] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299254996000, latest_log_ts:{val:1736326282603912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:24.519507] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.542734] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.543918] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.550969] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AB-0-0] [lt=90] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:24.551038] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AB-0-0] [lt=50] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326284317916001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326284317916001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:24.563622] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=93] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:24.573654] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58524][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D4D-0-0] [lt=26] init point thread id with(&point=0x5572d50e5040, idx_=1180, point=[thread id=58524, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58524) [2025-01-08 00:51:24.585387] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:24.585426] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.588375] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58526][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:24.588487] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:51:24.664405] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:24.665567] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58515][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=56] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:24.682260] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:24.682322] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44830, [delay/us]:prepare:7, schedule:97, submit:87, rt:2402, total:2594) [2025-01-08 00:51:24.682338] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:24.682398] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44830, [delay/us]: prepare:7, schedule:97, submit:87, device:2402, total:2594) [2025-01-08 00:51:24.682467] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=62] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:24.682660] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4527, iops:19, bw:85571, [delay/us]:prepare:12, schedule:78, submit:59, rt:2288, total:2439) [2025-01-08 00:51:24.682676] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:24.682728] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4527, avg_iops:19, avg_bw:85566, [delay/us]: prepare:12, schedule:78, submit:59, device:2288, total:2439) [2025-01-08 00:51:24.682745] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:24.682835] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20372, [delay/us]:prepare:6, schedule:296, submit:1, rt:2466, total:2771) [2025-01-08 00:51:24.682929] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:2, hold_mem:16384 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:171120, [delay/us]:prepare:5, schedule:110, submit:330, rt:2236, total:2683) [2025-01-08 00:51:24.682942] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1002, ref_cnt=6, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49998, free_result_cnt=49998, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:24.682974] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20370, [delay/us]: prepare:6, schedule:296, submit:1, device:2466, total:2771) [2025-01-08 00:51:24.682990] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:171109, [delay/us]: prepare:5, schedule:110, submit:330, device:2236, total:2683) [2025-01-08 00:51:24.683000] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:24.683021] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4074, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1907, total:1915) [2025-01-08 00:51:24.683044] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:24.683071] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4074, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1907, total:1915) [2025-01-08 00:51:24.683089] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:24.683103] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:24.683111] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:24.683119] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:24.683127] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=228kB/s, net_out=233kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:24.685661] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.689755] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=27] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:24.705511] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE33-0-0] [lt=16] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:24.705644] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE33-0-0] [lt=123] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:24.705681] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE33-0-0] [lt=34] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:24.712315] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=16] ====== tenant freeze timer task ====== [2025-01-08 00:51:24.748170] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=37] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:24.750341] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=93] get number(ret=0, service_type_=0, range=1, base_id=1736326284749327000, start_id=1736326284749327000, end_id=1736326284749327001) [2025-01-08 00:51:24.778772] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708515B-0-0] [lt=37] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:24.778953] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708515D-0-0] [lt=59] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:24.779445] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326284583412000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=196022, in_cluster_service=true, cluster_version={val:1736326284383131000, v:0}, min_cluster_version={val:1736326284383131000, v:0}, max_cluster_version={val:1736326284383131000, v:0}, get_cluster_version_err=0, cluster_version_delta=396303, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18818, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=630416217, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:24.785147] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:24.785364] WDIAG read_multi_row (ob_table_access_helper.h:238) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=110][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:24.785714] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.786982] INFO delete_row (ob_table_access_helper.h:339) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326284785380) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326284785380) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:24.793864] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58523][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D50-0-0] [lt=23] init point thread id with(&point=0x5572d50e5000, idx_=1179, point=[thread id=58523, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58523) [2025-01-08 00:51:24.794832] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43302042}], scn_array=[cnt:1, {val:1736326284781479000, v:0}], curr_size=589, accum_size=10372, time_cost=12184) [2025-01-08 00:51:24.796162] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=104] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:24.799877] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1002}, {id:1}]) [2025-01-08 00:51:24.800018] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:24.800146] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:51:24.800333] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=8] gc stale ls task succ [2025-01-08 00:51:24.801723] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326284801589], gts:1736326284800559000, latest_srr:[mts=1736326284801589]}) [2025-01-08 00:51:24.801822] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326284801754], gts:1736326284800559000, latest_srr:[mts=1736326284801754]}) [2025-01-08 00:51:24.802056] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326284802044], gts:1736326284800559000, latest_srr:[mts=1736326284802044]}) [2025-01-08 00:51:24.804591] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:219) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:24.804621] INFO [COMMON] handle (memory_dump.cpp:538) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=29] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:24.805676] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:24.805702] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:24.806277] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] update_check_range(min_check_version=374, max_check_version=375, global_version=376) [2025-01-08 00:51:24.817948] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918542F-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:24.818955] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58525][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86A4-0-0] [lt=27] init point thread id with(&point=0x5572d50e5080, idx_=1181, point=[thread id=58525, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58525) [2025-01-08 00:51:24.819484] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:320K/74:189821 disk:320K/74:189821 t1m2:40K/10:91756 t1m2g0:40K/10:91756 t1002m2:188K/44:165678 t1002m2g0:188K/44:165678 t1001m2:92K/20:291249 t1001m2g0:92K/20:291249 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:24.820564] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=41] ====== checkpoint timer task ====== [2025-01-08 00:51:24.822165] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=20] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:24.822340] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=173] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:24.822494] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=31] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299284252000, latest_log_ts:{val:1736326282955407001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:24.822584] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=85] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:24.822615] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:24.822630] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326282955407001, v:0}) [2025-01-08 00:51:24.822638] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:24.822647] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:24.822659] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:24.822677] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:24.822690] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:24.822711] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:24.822732] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:24.822743] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:24.828635] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=67] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:24.829970] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857C7-0-0] [lt=27] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=83) [2025-01-08 00:51:24.830306] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326284820404000, start_id=1736326284820404000, end_id=1736326284820404001) [2025-01-08 00:51:24.830533] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:24.830653] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:24.830815] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2199742}], scn_array=[cnt:1, {val:1736326284308664004, v:0}], curr_size=122, accum_size=967, time_cost=11334) [2025-01-08 00:51:24.830945] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2212272}], scn_array=[cnt:1, {val:1736326284317916004, v:0}], curr_size=122, accum_size=976, time_cost=11726) [2025-01-08 00:51:24.831107] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=40] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:24.836064] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870CA-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:24.836118] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:24.836130] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:24.841030] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:24.841354] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=2079744) [2025-01-08 00:51:24.845516] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=38] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:24.850727] INFO handle (memory_dump.cpp:665) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 152 261 0 1 2 68 1 0 1 4 1 1 0 1 5 43 5 0 1 7 1 2 0 1 8 48 1 0 1 12 7 3 0 1 13 1 1 0 1 14 1 2 0 1 15 1 1 0 1 16 1 1 0 1 18 4 3 0 1 20 1 2 0 500 0 94 240 0 500 7 7 4 0 500 8 37 1 0 500 9 2 1 0 500 10 3 2 0 500 16 2 1 0 500 17 9 7 0 500 22 79 61 0 500 23 23 10 0 508 0 2 10 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 98 230 0 1001 2 49 1 0 1001 4 1 1 0 1001 5 22 5 0 1001 8 30 1 0 1001 12 3 2 0 1001 13 1 1 0 1001 14 1 1 0 1001 18 4 3 0 1001 20 2 2 0 1002 0 162 245 0 1002 2 59 1 0 1002 4 2 1 0 1002 5 49 6 0 1002 7 11 2 0 1002 8 47 1 0 1002 12 2 3 0 1002 13 1 1 0 1002 14 1 1 0 1002 18 9 3 0 1002 20 1 2 0 cost_time: 44582 [2025-01-08 00:51:24.850845] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=40] [MEMORY] tenant: 1, limit: 2,147,483,648 hold: 1,044,885,504 cache_hold: 213,909,504 cache_used: 213,909,504 cache_item_count: 102 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 447,782,912 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 142,606,336 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 95,862,784 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 100,663,296 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 14,680,064 limit= 429,496,720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:24.851329] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 447,782,912 used= 344,099,520 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,211,520 used= 31,196,160 count= 15 avg_used= 2,079,744 block_cnt= 15 chunk_cnt= 15 mod=CACHE_MAP_NODE [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 26,599,424 used= 26,451,520 count= 24 avg_used= 1,102,146 block_cnt= 24 chunk_cnt= 18 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,196,912 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 37 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,268,096 used= 11,228,160 count= 4 avg_used= 2,807,040 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,618,560 used= 7,541,760 count= 10 avg_used= 754,176 block_cnt= 10 chunk_cnt= 8 mod=ResourceGroup [MEMORY] hold= 7,225,344 used= 6,209,536 count= 124 avg_used= 50,076 block_cnt= 124 chunk_cnt= 23 mod=[T]char [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMemCU [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMem [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,440,640 used= 3,015,040 count= 140 avg_used= 21,536 block_cnt= 140 chunk_cnt= 29 mod=DiagnosticInfo [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,780,352 used= 2,663,280 count= 586 avg_used= 4,544 block_cnt= 570 chunk_cnt= 29 mod=WorkerMap [MEMORY] hold= 2,768,896 used= 2,720,000 count= 13 avg_used= 209,230 block_cnt= 13 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,640 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 272 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,394,368 used= 1,372,072 count= 75 avg_used= 18,294 block_cnt= 75 chunk_cnt= 11 mod=TabletMap [MEMORY] hold= 1,374,720 used= 1,308,288 count= 36 avg_used= 36,341 block_cnt= 36 chunk_cnt= 13 mod=CommSysVarFac [MEMORY] hold= 1,327,104 used= 1,322,784 count= 9 avg_used= 146,976 block_cnt= 9 chunk_cnt= 7 mod=SQLSessionInfo [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 960,192 used= 936,448 count= 102 avg_used= 9,180 block_cnt= 102 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 933,888 used= 458,736 count= 57 avg_used= 8,048 block_cnt= 57 chunk_cnt= 19 mod=[T]ObPerfEventR [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 511,488 used= 490,752 count= 108 avg_used= 4,544 block_cnt= 108 chunk_cnt= 25 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 262,144 used= 150,272 count= 16 avg_used= 9,392 block_cnt= 16 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 262,144 used= 261,120 count= 4 avg_used= 65,280 block_cnt= 4 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 203,840 used= 194,432 count= 49 avg_used= 3,968 block_cnt= 49 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 12 mod=SqlSession [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 13 mod=LinearHashMapNo [MEMORY] hold= 155,024 used= 2,177 count= 1,920 avg_used= 1 block_cnt= 19 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 154,432 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 13 mod=QueryAllocator [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,456 used= 131,072 count= 2 avg_used= 65,536 block_cnt= 2 chunk_cnt= 2 mod=MinorMergeMgr [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 105,664 used= 103,168 count= 13 avg_used= 7,936 block_cnt= 13 chunk_cnt= 8 mod=SqlSessiVarMap [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 7 mod=PlanVaIdx [MEMORY] hold= 93,312 used= 24,208 count= 343 avg_used= 70 block_cnt= 78 chunk_cnt= 27 mod=Coro [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 5 mod=HashNodeLCSta [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 2 mod=HB_SERVICE [MEMORY] hold= 54,720 used= 51,840 count= 15 avg_used= 3,456 block_cnt= 12 chunk_cnt= 11 mod=CACHE_INST [MEMORY] hold= 52,736 used= 40,992 count= 61 avg_used= 672 block_cnt= 60 chunk_cnt= 21 mod=[T]ObWarningBuf [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 40,400 used= 28,224 count= 63 avg_used= 448 block_cnt= 33 chunk_cnt= 12 mod=OMT_Worker [MEMORY] hold= 38,048 used= 4,032 count= 168 avg_used= 24 block_cnt= 69 chunk_cnt= 28 mod=[T]MemoryContex [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PartCtx [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,640 used= 15,872 count= 4 avg_used= 3,968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnMemNode [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,640 used= 6,320 count= 12 avg_used= 526 block_cnt= 12 chunk_cnt= 8 mod=SqlSessiQuerSql [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=SqlSessionSbloc [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 7,504 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 7 mod=PackStateMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 5 mod=SequenceIdMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 6 mod=SequenceMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 8 mod=ContextsMap [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,848 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 5 mod=DICollector [MEMORY] hold= 4,864 used= 4,288 count= 3 avg_used= 1,429 block_cnt= 2 chunk_cnt= 2 mod=NegParamIdex [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 2 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSIDDLVar [MEMORY] hold= 344,099,520 used= 336,478,882 count= 31,610 avg_used= 10,644 mod=SUMMARY [2025-01-08 00:51:24.851393] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=49] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 142,606,336 used= 141,492,224 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 block_cnt= 68 chunk_cnt= 68 mod=Memstore [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:24.851414] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:24.851449] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 95,862,784 used= 90,371,200 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83,884,032 used= 75,811,934 count= 2,908 avg_used= 26,070 block_cnt= 2,908 chunk_cnt= 43 mod=SqlPhyPlan [MEMORY] hold= 4,709,056 used= 4,468,656 count= 313 avg_used= 14,276 block_cnt= 313 chunk_cnt= 41 mod=SqlPlanCache [MEMORY] hold= 1,353,472 used= 1,234,048 count= 622 avg_used= 1,984 block_cnt= 208 chunk_cnt= 39 mod=CreateContext [MEMORY] hold= 422,656 used= 412,672 count= 52 avg_used= 7,936 block_cnt= 52 chunk_cnt= 12 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 90,371,200 used= 81,928,910 count= 3,897 avg_used= 21,023 mod=SUMMARY [2025-01-08 00:51:24.851479] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,368 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,272 used= 58,751 count= 259 avg_used= 226 block_cnt= 21 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,368 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-01-08 00:51:24.851502] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 100,663,296 used= 97,542,144 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97,542,144 used= 97,360,704 count= 189 avg_used= 515,136 block_cnt= 189 chunk_cnt= 48 mod=CoStack [MEMORY] hold= 97,542,144 used= 97,360,704 count= 189 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:24.851529] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 213,909,504 used= 213,909,504 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:24.851547] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 14,680,064 used= 12,713,920 limit= 429,496,720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,036,352 used= 7,149,419 count= 109 avg_used= 65,591 block_cnt= 109 chunk_cnt= 5 mod=L_TabletPool [MEMORY] hold= 4,587,520 used= 4,569,600 count= 70 avg_used= 65,280 block_cnt= 70 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 12,713,920 used= 11,803,787 count= 181 avg_used= 65,214 mod=SUMMARY [2025-01-08 00:51:24.851561] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,584,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 block_cnt= 195 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:24.851577] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 16,256 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ExtInfoLog [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:24.851590] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 33,216 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 mod=SUMMARY [2025-01-08 00:51:24.851604] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 2,097,152 used= 4,160 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4,160 used= 3,968 count= 1 avg_used= 3,968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_GET_GTS_R [MEMORY] hold= 4,160 used= 3,968 count= 1 avg_used= 3,968 mod=SUMMARY [2025-01-08 00:51:24.851624] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 14,017,536 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 9,473,984 used= 9,445,184 count= 3 avg_used= 3,148,394 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 4,535,424 used= 4,428,288 count= 558 avg_used= 7,936 block_cnt= 558 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 14,017,536 used= 13,881,408 count= 562 avg_used= 24,700 mod=SUMMARY [2025-01-08 00:51:24.851643] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 73,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 73,152 used= 71,424 count= 9 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:24.851678] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 633,344,000 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 232,894,464 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 77,594,624 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 24,260,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 170,676,224 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98,557,952 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:24.851988] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 232,894,464 used= 186,149,728 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,397,376 used= 7,340,256 count= 7 avg_used= 1,048,608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7,259,504 used= 6,324,350 count= 1,032 avg_used= 6,128 block_cnt= 540 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,783,552 used= 5,752,960 count= 6 avg_used= 958,826 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 16 mod=Iteratorget_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:24.853314] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 314,245,120 used= 251,735,872 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 15 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,646,144 used= 16,637,952 count= 8 avg_used= 2,079,744 block_cnt= 8 chunk_cnt= 8 mod=CACHE_MAP_NODE [MEMORY] hold= 16,196,944 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 41 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 16,195,584 used= 16,052,800 count= 19 avg_used= 844,884 block_cnt= 19 chunk_cnt= 13 mod=MysqlRequesReco [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 4,214,784 used= 4,194,336 count= 1 avg_used= 4,194,336 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 17 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 16 mod=CkptDgnMemCU [MEMORY] hold= 3,809,280 used= 3,770,880 count= 5 avg_used= 754,176 block_cnt= 5 chunk_cnt= 4 mod=ResourceGroup [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,686,400 used= 3,211,264 count= 58 avg_used= 55,366 block_cnt= 58 chunk_cnt= 16 mod=[T]char [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,933,312 used= 1,914,880 count= 3 avg_used= 638,293 block_cnt= 3 chunk_cnt= 2 mod=ServerObjecPool [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,337,616 used= 1,316,664 count= 68 avg_used= 19,362 block_cnt= 68 chunk_cnt= 10 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,092,032 used= 1,046,832 count= 226 avg_used= 4,632 block_cnt= 226 chunk_cnt= 16 mod=WorkerMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=DiagnosticInfos [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 917,504 used= 870,400 count= 8 avg_used= 108,800 block_cnt= 8 chunk_cnt= 6 mod=DiagnosticSum [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 3 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 458,176 used= 440,550 count= 6 avg_used= 73,425 block_cnt= 6 chunk_cnt= 4 mod=SqlMemMgr [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 245,760 used= 120,720 count= 15 avg_used= 8,048 block_cnt= 15 chunk_cnt= 9 mod=[T]ObPerfEventR [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 196,608 used= 195,840 count= 3 avg_used= 65,280 block_cnt= 3 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 187,200 used= 178,560 count= 45 avg_used= 3,968 block_cnt= 45 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 180,224 used= 103,312 count= 11 avg_used= 9,392 block_cnt= 11 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,456 used= 146,976 count= 1 avg_used= 146,976 block_cnt= 1 chunk_cnt= 1 mod=SQLSessionInfo [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 7 mod=QueryAllocator [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 8 mod=LinearHashMapNo [MEMORY] hold= 89,408 used= 87,296 count= 11 avg_used= 7,936 block_cnt= 11 chunk_cnt= 5 mod=ParamCharsType [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 7 mod=TabletIDSetNode [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 71,040 used= 68,160 count= 15 avg_used= 4,544 block_cnt= 15 chunk_cnt= 10 mod=[T]ObTraceEvent [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 59,904 used= 14,880 count= 224 avg_used= 66 block_cnt= 57 chunk_cnt= 22 mod=Coro [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=MdsTableMgr [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 3 mod=HashNodeLCSta [MEMORY] hold= 29,184 used= 27,648 count= 8 avg_used= 3,456 block_cnt= 8 chunk_cnt= 6 mod=CACHE_INST [MEMORY] hold= 28,352 used= 23,232 count= 2 avg_used= 11,616 block_cnt= 2 chunk_cnt= 2 mod=DI_CONTAINER [MEMORY] hold= 25,760 used= 2,736 count= 114 avg_used= 24 block_cnt= 44 chunk_cnt= 17 mod=[T]MemoryContex [MEMORY] hold= 25,552 used= 19,232 count= 2 avg_used= 9,616 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 21,536 count= 1 avg_used= 21,536 block_cnt= 1 chunk_cnt= 1 mod=DiagnosticInfo [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 9,336 count= 1 avg_used= 9,336 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_AssignedMgr [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,000 used= 15,616 count= 2 avg_used= 7,808 block_cnt= 2 chunk_cnt= 2 mod=HashNodePlanCac [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 14,720 used= 10,304 count= 23 avg_used= 448 block_cnt= 11 chunk_cnt= 5 mod=OMT_Worker [MEMORY] hold= 13,824 used= 10,752 count= 16 avg_used= 672 block_cnt= 15 chunk_cnt= 10 mod=[T]ObWarningBuf [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 2 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PartCtx [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,880 used= 1,920 count= 5 avg_used= 384 block_cnt= 4 chunk_cnt= 3 mod=TableLockNode [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,496 used= 1,920 count= 3 avg_used= 640 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 2 chunk_cnt= 2 mod=DICollector [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 1 chunk_cnt= 1 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 288 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 251,735,872 used= 247,943,519 count= 3,083 avg_used= 80,422 mod=SUMMARY [2025-01-08 00:51:24.853393] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=60] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 102,760,448 used= 101,957,632 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 101,957,632 used= 101,910,984 count= 49 avg_used= 2,079,816 block_cnt= 49 chunk_cnt= 49 mod=Memstore [MEMORY] hold= 101,957,632 used= 101,910,984 count= 49 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:24.853415] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:24.853523] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=88] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 46,137,344 used= 30,700,864 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 28,643,136 used= 25,577,143 count= 1,233 avg_used= 20,743 block_cnt= 1,233 chunk_cnt= 22 mod=SqlPhyPlan [MEMORY] hold= 1,172,416 used= 1,085,952 count= 125 avg_used= 8,687 block_cnt= 125 chunk_cnt= 19 mod=SqlPlanCache [MEMORY] hold= 574,464 used= 523,776 count= 264 avg_used= 1,984 block_cnt= 88 chunk_cnt= 18 mod=CreateContext [MEMORY] hold= 308,864 used= 301,568 count= 38 avg_used= 7,936 block_cnt= 38 chunk_cnt= 7 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 30,700,864 used= 27,490,039 count= 1,662 avg_used= 16,540 mod=SUMMARY [2025-01-08 00:51:24.853549] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 62,914,560 used= 59,867,136 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 59,867,136 used= 59,755,776 count= 116 avg_used= 515,136 block_cnt= 116 chunk_cnt= 30 mod=CoStack [MEMORY] hold= 59,867,136 used= 59,755,776 count= 116 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:24.853569] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 79,691,776 used= 79,691,776 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 79,691,776 used= 0 count= 38 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 79,691,776 used= 0 count= 38 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:24.853584] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 6,291,456 used= 4,153,280 limit= 214,748,360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4,063,232 used= 4,047,360 count= 62 avg_used= 65,280 block_cnt= 62 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 4,153,280 used= 4,132,128 count= 64 avg_used= 64,564 mod=SUMMARY [2025-01-08 00:51:24.853597] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,064,768 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,064,768 used= 1,039,616 count= 131 avg_used= 7,936 block_cnt= 131 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,064,768 used= 1,039,616 count= 131 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:24.853682] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:24.853732] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=36] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 2,097,152 used= 0 limit= 1,073,741,800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-01-08 00:51:24.853762] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 23,089,152 used= 22,820,672 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 17,862,592 used= 17,833,792 count= 3 avg_used= 5,944,597 block_cnt= 3 chunk_cnt= 2 mod=MEMTABLE_ARENA [MEMORY] hold= 4,949,952 used= 4,833,024 count= 609 avg_used= 7,936 block_cnt= 609 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 22,820,672 used= 22,674,752 count= 613 avg_used= 36,989 mod=SUMMARY [2025-01-08 00:51:24.853852] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 4,194,304 used= 384,320 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 359,936 used= 351,088 count= 10 avg_used= 35,108 block_cnt= 10 chunk_cnt= 2 mod=TX_OP [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 384,320 used= 374,896 count= 13 avg_used= 28,838 mod=SUMMARY [2025-01-08 00:51:24.854026] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=74] [MEMORY] tenant: 1002, limit: 3,221,225,472 hold: 1,114,390,528 cache_hold: 52,428,800 cache_used: 52,428,800 cache_item_count: 25 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 596,312,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 123,731,968 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 102,760,448 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 23,068,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 98,566,144 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 4,194,304 limit= 644,245,080 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 69,287,936 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33,554,432 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:24.855297] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D065-0-0] [lt=23][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:24.855781] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=20] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:24.856308] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1736326284856305, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000745, add_bytes: 0 [2025-01-08 00:51:24.857523] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=22] get number(ret=0, service_type_=0, range=1, base_id=1736326284855640000, start_id=1736326284855640000, end_id=1736326284855640001) [2025-01-08 00:51:24.862795] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84336423}], scn_array=[cnt:1, {val:1736326284820404000, v:0}], curr_size=545, accum_size=3246, time_cost=30367) [2025-01-08 00:51:24.865051] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 596,312,064 used= 507,009,936 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 30 [MEMORY] hold= 167,854,080 used= 167,772,160 count= 4 avg_used= 41,943,040 block_cnt= 4 chunk_cnt= 4 mod=LogGroupBuffer [MEMORY] hold= 82,780,160 used= 82,604,608 count= 51 avg_used= 1,619,698 block_cnt= 51 chunk_cnt= 43 mod=MysqlRequesReco [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 7 mod=IoControl [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,646,144 used= 16,637,952 count= 8 avg_used= 2,079,744 block_cnt= 8 chunk_cnt= 8 mod=CACHE_MAP_NODE [MEMORY] hold= 16,196,944 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 41 chunk_cnt= 7 mod=OmtTenant [MEMORY] hold= 14,893,056 used= 14,794,816 count= 8 avg_used= 1,849,352 block_cnt= 8 chunk_cnt= 8 mod=LogDIOAligned [MEMORY] hold= 13,975,552 used= 13,926,400 count= 3 avg_used= 4,642,133 block_cnt= 3 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 6,856,704 used= 6,787,584 count= 9 avg_used= 754,176 block_cnt= 9 chunk_cnt= 9 mod=ResourceGroup [MEMORY] hold= 6,340,608 used= 5,505,024 count= 102 avg_used= 53,970 block_cnt= 102 chunk_cnt= 19 mod=[T]char [MEMORY] hold= 4,943,872 used= 4,915,456 count= 2 avg_used= 2,457,728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 20 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 20 mod=CkptDgnMemCU [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 6 mod=SqlDtlQueue [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,831,808 used= 2,713,104 count= 594 avg_used= 4,567 block_cnt= 583 chunk_cnt= 21 mod=WorkerMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,326,528 used= 2,284,800 count= 11 avg_used= 207,709 block_cnt= 11 chunk_cnt= 9 mod=DiagnosticSum [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,978,176 used= 1,960,616 count= 50 avg_used= 39,212 block_cnt= 50 chunk_cnt= 7 mod=TabletMap [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,605,632 used= 1,574,016 count= 4 avg_used= 393,504 block_cnt= 4 chunk_cnt= 4 mod=ClogGe [MEMORY] hold= 1,376,256 used= 1,343,488 count= 4 avg_used= 335,872 block_cnt= 4 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 1,335,296 used= 1,331,072 count= 1 avg_used= 1,331,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=DiagnosticInfos [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 870,848 used= 849,152 count= 96 avg_used= 8,845 block_cnt= 96 chunk_cnt= 16 mod=ParamCharsType [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 3 chunk_cnt= 3 mod=PartitTableTask [MEMORY] hold= 787,264 used= 524,352 count= 36 avg_used= 14,565 block_cnt= 34 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 720,896 used= 354,112 count= 44 avg_used= 8,048 block_cnt= 44 chunk_cnt= 17 mod=[T]ObPerfEventR [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 589,824 used= 587,904 count= 4 avg_used= 146,976 block_cnt= 4 chunk_cnt= 3 mod=SQLSessionInfo [MEMORY] hold= 491,520 used= 442,432 count= 8 avg_used= 55,304 block_cnt= 8 chunk_cnt= 5 mod=TabletIDSetBkt [MEMORY] hold= 466,880 used= 451,312 count= 3 avg_used= 150,437 block_cnt= 3 chunk_cnt= 3 mod=IndexUsageMgr [MEMORY] hold= 458,752 used= 444,384 count= 4 avg_used= 111,096 block_cnt= 4 chunk_cnt= 3 mod=AsyncFreezeTabl [MEMORY] hold= 450,272 used= 432,838 count= 5 avg_used= 86,567 block_cnt= 5 chunk_cnt= 4 mod=SqlMemMgr [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 393,216 used= 372,096 count= 4 avg_used= 93,024 block_cnt= 4 chunk_cnt= 4 mod=LSSvr [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 1 mod=bf_queue [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 278,528 used= 159,664 count= 17 avg_used= 9,392 block_cnt= 17 chunk_cnt= 9 mod=[T]ObDSActionAr [MEMORY] hold= 270,336 used= 246,624 count= 3 avg_used= 82,208 block_cnt= 3 chunk_cnt= 2 mod=HTblRkQueAlloc [MEMORY] hold= 229,376 used= 221,688 count= 1 avg_used= 221,688 block_cnt= 1 chunk_cnt= 1 mod=IndexUsageRepor [MEMORY] hold= 229,376 used= 203,776 count= 4 avg_used= 50,944 block_cnt= 4 chunk_cnt= 3 mod=palf_env [MEMORY] hold= 213,120 used= 204,480 count= 45 avg_used= 4,544 block_cnt= 45 chunk_cnt= 15 mod=[T]ObTraceEvent [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 163,840 used= 155,664 count= 2 avg_used= 77,832 block_cnt= 2 chunk_cnt= 2 mod=IteratorStorage [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 137,280 used= 130,944 count= 33 avg_used= 3,968 block_cnt= 33 chunk_cnt= 7 mod=TinyAllocator [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=LockMemObj [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98,304 used= 86,144 count= 4 avg_used= 21,536 block_cnt= 4 chunk_cnt= 2 mod=DiagnosticInfo [MEMORY] hold= 91,904 used= 23,824 count= 338 avg_used= 70 block_cnt= 83 chunk_cnt= 26 mod=Coro [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81,280 used= 79,360 count= 10 avg_used= 7,936 block_cnt= 10 chunk_cnt= 6 mod=QueryAllocator [MEMORY] hold= 81,280 used= 79,360 count= 10 avg_used= 7,936 block_cnt= 10 chunk_cnt= 8 mod=LinearHashMapNo [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=MinorMergeMgr [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 46,336 count= 4 avg_used= 11,584 block_cnt= 4 chunk_cnt= 4 mod=LogApplyStatus [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 40,992 count= 4 avg_used= 10,248 block_cnt= 4 chunk_cnt= 4 mod=TabletSvrBucket [MEMORY] hold= 65,536 used= 56,064 count= 4 avg_used= 14,016 block_cnt= 4 chunk_cnt= 3 mod=LogReplayStatus [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 6 mod=HashNodeLCSta [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 42,064 used= 4,464 count= 186 avg_used= 24 block_cnt= 63 chunk_cnt= 24 mod=[T]MemoryContex [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 38,400 used= 26,880 count= 60 avg_used= 448 block_cnt= 33 chunk_cnt= 14 mod=OMT_Worker [MEMORY] hold= 38,016 used= 29,568 count= 44 avg_used= 672 block_cnt= 43 chunk_cnt= 15 mod=[T]ObWarningBuf [MEMORY] hold= 32,768 used= 31,744 count= 2 avg_used= 15,872 block_cnt= 2 chunk_cnt= 2 mod=NotParamInfo [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MatchLsnMap [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 31,936 used= 26,816 count= 2 avg_used= 13,408 block_cnt= 2 chunk_cnt= 2 mod=DI_CONTAINER [MEMORY] hold= 29,184 used= 27,648 count= 8 avg_used= 3,456 block_cnt= 5 chunk_cnt= 5 mod=CACHE_INST [MEMORY] hold= 28,992 used= 27,792 count= 6 avg_used= 4,632 block_cnt= 6 chunk_cnt= 3 mod=TTLTaskBucket [MEMORY] hold= 25,728 used= 24,576 count= 6 avg_used= 4,096 block_cnt= 6 chunk_cnt= 5 mod=LinearHashMapCn [MEMORY] hold= 25,728 used= 24,576 count= 6 avg_used= 4,096 block_cnt= 6 chunk_cnt= 6 mod=LinearHashMapDi [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=PartCtx [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ObDataDictSvc [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ALLTTLTaskSche [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,616 used= 7,424 count= 1 avg_used= 7,424 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,976 used= 6,176 count= 4 avg_used= 1,544 block_cnt= 4 chunk_cnt= 3 mod=TXBFRetryId [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 3,968 used= 3,200 count= 4 avg_used= 800 block_cnt= 4 chunk_cnt= 3 mod=DICollector [MEMORY] hold= 3,904 used= 3,104 count= 4 avg_used= 776 block_cnt= 4 chunk_cnt= 3 mod=DepTabletSet [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 3,136 used= 2,560 count= 3 avg_used= 853 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 2,752 used= 2,176 count= 3 avg_used= 725 block_cnt= 3 chunk_cnt= 3 mod=NotParamIdex [MEMORY] hold= 2,688 used= 1,888 count= 4 avg_used= 472 block_cnt= 4 chunk_cnt= 3 mod=Election [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,304 used= 1,344 count= 5 avg_used= 268 block_cnt= 4 chunk_cnt= 4 mod=NegParamIdex [MEMORY] hold= 2,016 used= 864 count= 6 avg_used= 144 block_cnt= 5 chunk_cnt= 5 mod=ObTxLSLogCb [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,888 used= 256 count= 8 avg_used= 32 block_cnt= 6 chunk_cnt= 4 mod=PalfFSCbNode [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,216 used= 416 count= 4 avg_used= 104 block_cnt= 4 chunk_cnt= 3 mod=RedoLogBuckLock [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 960 used= 164 count= 4 avg_used= 41 block_cnt= 4 chunk_cnt= 4 mod=KeepAliveBuf [MEMORY] hold= 896 used= 128 count= 4 avg_used= 32 block_cnt= 3 chunk_cnt= 3 mod=PalfRCCbNode [MEMORY] hold= 896 used= 128 count= 4 avg_used= 32 block_cnt= 4 chunk_cnt= 4 mod=RebuildCbNode [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 640 used= 440 count= 1 avg_used= 440 block_cnt= 1 chunk_cnt= 1 mod=tenant_ttl_mgr [MEMORY] hold= 608 used= 204 count= 2 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=TableLockNode [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 507,009,936 used= 501,868,424 count= 4,029 avg_used= 124,564 mod=SUMMARY [2025-01-08 00:51:24.865127] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=56] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 123,731,968 used= 122,765,312 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 122,765,312 used= 122,709,144 count= 59 avg_used= 2,079,816 block_cnt= 59 chunk_cnt= 59 mod=Memstore [MEMORY] hold= 122,765,312 used= 122,709,144 count= 59 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:24.865149] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 4,194,304 used= 2,916,352 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,916,352 used= 2,890,240 count= 4 avg_used= 722,560 block_cnt= 4 chunk_cnt= 2 mod=PartTranCtxMgr [MEMORY] hold= 2,916,352 used= 2,890,240 count= 4 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:24.865171] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 102,760,448 used= 99,366,720 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 91,521,024 used= 82,598,964 count= 3,688 avg_used= 22,396 block_cnt= 3,688 chunk_cnt= 49 mod=SqlPhyPlan [MEMORY] hold= 4,355,904 used= 4,089,488 count= 336 avg_used= 12,171 block_cnt= 336 chunk_cnt= 47 mod=SqlPlanCache [MEMORY] hold= 1,475,136 used= 1,369,760 count= 155 avg_used= 8,837 block_cnt= 155 chunk_cnt= 2 mod=SqlPhyPlObj [MEMORY] hold= 1,427,456 used= 1,301,504 count= 656 avg_used= 1,984 block_cnt= 219 chunk_cnt= 48 mod=CreateContext [MEMORY] hold= 585,216 used= 571,392 count= 72 avg_used= 7,936 block_cnt= 72 chunk_cnt= 11 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 99,366,720 used= 89,932,708 count= 4,909 avg_used= 18,319 mod=SUMMARY [2025-01-08 00:51:24.865192] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 23,068,672 used= 22,016,304 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 22,002,832 used= 19,331,822 count= 32,760 avg_used= 590 block_cnt= 1,482 chunk_cnt= 11 mod=PlJit [MEMORY] hold= 13,472 used= 13,344 count= 2 avg_used= 6,672 block_cnt= 2 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 22,016,304 used= 19,345,166 count= 32,762 avg_used= 590 mod=SUMMARY [2025-01-08 00:51:24.865206] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 98,566,144 used= 95,993,856 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 95,993,856 used= 95,815,296 count= 186 avg_used= 515,136 block_cnt= 186 chunk_cnt= 47 mod=CoStack [MEMORY] hold= 95,993,856 used= 95,815,296 count= 186 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:24.865227] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 52,428,800 used= 52,428,800 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 52,428,800 used= 0 count= 25 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 52,428,800 used= 0 count= 25 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:24.865378] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 4,194,304 used= 3,178,432 limit= 644,245,080 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,014,656 used= 3,002,880 count= 46 avg_used= 65,280 block_cnt= 46 chunk_cnt= 2 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 73,728 used= 65,591 count= 1 avg_used= 65,591 block_cnt= 1 chunk_cnt= 1 mod=L_TabletPool [MEMORY] hold= 3,178,432 used= 3,153,239 count= 49 avg_used= 64,351 mod=SUMMARY [2025-01-08 00:51:24.865401] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 755,904 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 755,904 used= 738,048 count= 93 avg_used= 7,936 block_cnt= 93 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 755,904 used= 738,048 count= 93 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:24.865416] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:24.865444] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 69,287,936 used= 63,614,464 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 63,061,760 used= 62,946,560 count= 12 avg_used= 5,245,546 block_cnt= 12 chunk_cnt= 9 mod=MEMTABLE_ARENA [MEMORY] hold= 520,192 used= 507,904 count= 64 avg_used= 7,936 block_cnt= 64 chunk_cnt= 1 mod=TX_DATA_SLICE [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 63,614,464 used= 63,486,208 count= 80 avg_used= 793,577 mod=SUMMARY [2025-01-08 00:51:24.865463] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 65,024 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:24.865486] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33,554,432 used= 0 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 33,554,432 free_size= 33,554,432 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-01-08 00:51:24.868176] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43303220}], scn_array=[cnt:1, {val:1736326284855640000, v:0}], curr_size=546, accum_size=11507, time_cost=10405) [2025-01-08 00:51:24.869695] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:24.870258] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA3-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:24.871375] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=880796, clean_num=31457) [2025-01-08 00:51:24.871413] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:24.871495] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=37] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:24.880365] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDD-0-0] [lt=81] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:24.880419] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDD-0-0] [lt=45] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=56) [2025-01-08 00:51:24.883376] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=41] [APPLY STAT CB APPEND COST TIME](cur_stat_count=38, stat_interval=5000000, avg (count/cost)=113, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:24.883478] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=99] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=38, stat_interval=5000000, avg (count/cost)=10768, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:24.883495] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=17] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=38, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:24.883506] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB EXECUTE TIME](cur_stat_count=38, stat_interval=5000000, avg (count/cost)=-948, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:24.883517] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB TOTAL TIME](cur_stat_count=38, stat_interval=5000000, avg (count/cost)=9933, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:24.886160] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.888499] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA3-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:24.896787] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.896784] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=129] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.896890] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.896945] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:24.903651] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=80, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:24.903677] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:24.904072] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=19] ====== checkpoint timer task ====== [2025-01-08 00:51:24.905139] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=209] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:24.905271] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=131] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:24.905411] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=100] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2645, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:24.905448] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=551, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:24.905497] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=61] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:24.905373] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299254996000, latest_log_ts:{val:1736326282603912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:24.905535] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=158] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299254996000, latest_log_ts:{val:1736326282603912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:24.905598] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=57] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:24.905612] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] get rec log scn(service_type_=0, rec_log_ts={val:1736326282603912001, v:0}) [2025-01-08 00:51:24.905619] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:24.905625] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:24.905638] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:24.905655] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:24.905728] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=69] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:24.905754] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:24.905775] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:24.905760] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2BC-0-0] [lt=7] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326284915654, keepalive_ts:-1}) [2025-01-08 00:51:24.905785] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:24.905788] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2BC-0-0] [lt=27] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326284915654, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:24.906371] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2BB-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326284915486, keepalive_ts:-1}) [2025-01-08 00:51:24.906394] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2BB-0-0] [lt=23] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326284915486, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:24.908349] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A92D9-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:51:24.909376] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58478][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F6-0-0] [lt=11] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326284916381, keepalive_ts:-1}) [2025-01-08 00:51:24.909429] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58478][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F6-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326284916381, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:24.910238] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58522][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2D-0-0] [lt=26] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326284919939, keepalive_ts:-1}) [2025-01-08 00:51:24.910279] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58522][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2D-0-0] [lt=40] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326284919939, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:24.912459] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF57-0-0] [lt=24] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326285021088, keepalive_ts:-1}) [2025-01-08 00:51:24.912532] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF57-0-0] [lt=72] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326285021088, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:24.935679] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C666-0-0] [lt=85][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193789849510, cache_obj->added_lc()=false, cache_obj->get_object_id()=13633, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:24.957113] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=36] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12806676480, fast_delete_log_mode=false) [2025-01-08 00:51:24.966867] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=64] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:24.970140] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4763, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:24.970248] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=397, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:24.970330] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=91] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=23, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:24.971230] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58527][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:24.971281] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:51:24.972083] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C1D-0-0] [lt=130] init point thread id with(&point=0x5572d50e5100, idx_=1183, point=[thread id=58527, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58527) [2025-01-08 00:51:24.977545] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=106] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=21, avg_log_batch_cnt=1, total_group_log_size=11629, avg_group_log_size=553) [2025-01-08 00:51:24.977637] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=83] [PALF STAT APPEND COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:24.980939] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58522][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA39EC-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11629) [2025-01-08 00:51:24.981900] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=68] no need to update min/max version(new_version={val:1736326284583412000, v:0}, min_version={val:1736326284583412000, v:0}, max_version={val:1736326284583412000, v:0}) [2025-01-08 00:51:24.982786] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=114] [PALF STAT WRITE LOG COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=3004, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:24.982814] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=11629) [2025-01-08 00:51:24.982891] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=69] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=202, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:24.982936] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=37, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:24.982953] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT LOG LIFE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=8804, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:24.982962] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=21, avg_log_gen_to_freeze_time=646, avg_log_gen_to_submit_time=731, avg_log_submit_to_flush_time=4215, avg_log_submit_to_slide_time=8072) [2025-01-08 00:51:24.983044] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=70] [PALF STAT FLUSH CB COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=37, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:24.983231] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=11629, hit_cnt=21, read_cnt=21, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:24.984606] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=86] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326280764668, last_first_register_time_us=1736324449183670, register_time_us=1736326280764668, curr_time_us=1736326284984603) [2025-01-08 00:51:24.984743] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=84] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326284984691, last_submit_register_req_time_us=1736326284984691, register_parent_reason=1) [2025-01-08 00:51:24.984763] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:24.985867] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=70, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:24.985917] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1521, avg_group_log_size=169) [2025-01-08 00:51:24.985936] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1521) [2025-01-08 00:51:24.985948] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:24.985999] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=42, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:24.986033] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:24.986052] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1098) [2025-01-08 00:51:24.986067] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:24.986337] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=127][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:24.989552] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86A6-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:24.991048] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86A7-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1521) [2025-01-08 00:51:24.993556] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3247, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:24.993584] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:24.993617] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=669, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:24.993644] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1521) [2025-01-08 00:51:24.993665] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:24.993687] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6754, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:24.993719] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=27, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:24.993748] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:24.993757] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=5034, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:24.993803] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:24.994050] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=290] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=100, avg_log_submit_to_flush_time=3658, avg_log_submit_to_slide_time=4934) [2025-01-08 00:51:24.994144] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1521, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:24.995743] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=58] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326284297771003, v:0}, new_max_version={val:1736326284297771003, v:0}, last_min_version={val:1736326284297771001, v:0}, last_max_version={val:1736326284297771001, v:0}, total_time_us=8100, delta_us=697969) [2025-01-08 00:51:24.995777] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326284297771003, v:0}, min_version={val:1736326284297771003, v:0}, max_version={val:1736326284297771003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:24.999594] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4661-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:24.999653] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4661-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:24.999670] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4661-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:25.001522] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2515, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:25.001548] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:25.001589] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1348, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:25.001688] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=62, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:25.003984] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3901-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:25.004016] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3901-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:25.004700] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86A8-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:25.004723] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86A8-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22516, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:25.004732] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86A8-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=15, avg_log_submit_to_flush_time=4057, avg_log_submit_to_slide_time=22501) [2025-01-08 00:51:25.004825] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:25.004944] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18086, miss_cnt:0, cache_read_size:2212638, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:25.005002] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=43] succ to submit log task to replay service(to_submit_lsn={lsn:2212572}, to_submit_scn={val:1736326284830537001, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326285004669, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2212638}, next_to_submit_scn_:{val:1736326284830537002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2212516}, end_lsn:{lsn:2212638}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2212638}, readable_log_tail:{lsn:2212638}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64892130, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326284830537001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326284830537001, v:0}, data_checksum:3348837161, flag:62620}}, init_mode_version:0, accumulate_checksum:2451910402, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326284830537001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18086, miss_cnt:0, cache_read_size:2212638, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:25.006686] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:25.063993] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5255593176, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:25.064087] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:25.064102] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8ac3f7ef0) [2025-01-08 00:51:25.064111] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5255593176, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:25.071614] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:25.071891] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:25.086502] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.088412] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2966, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:25.088442] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=285, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:25.088575] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=58, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:25.094254] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:32779, max_lsn:43305066, max_scn:{val:1736326285084561000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:43305066}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:1362026}}, last_submit_lsn:{lsn:43304477}, last_submit_end_lsn:{lsn:43305066}, last_submit_log_id:32779, last_submit_log_pid:1, max_flushed_lsn:{lsn:43304477}, max_flushed_end_lsn:{lsn:43305066}, max_flushed_log_pid:1, committed_end_lsn:{lsn:43305066}, last_slide_log_id:32779, last_slide_scn:{val:1736326285084561000, v:0}, last_slide_lsn:{lsn:43304477}, last_slide_end_lsn:{lsn:43305066}, last_slide_log_pid:1, last_slide_log_accum_checksum:1478346714, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad9083ee530}, [StateMgr]={this:0x2ad9083f2830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:32780, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324348477669, palf_role_change_cb:0x2ad9083f92f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad9083efc70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324348678844, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/meta", dir_fd:172, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad9083f4a38, flashback_version:0}, log_storage:{log_tail:{lsn:43305066}, readable_log_tail:{lsn:43305066}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23799702, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324348477523, this:0x2ad9083f2f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.138:2882";"192.168.159.139:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324348478003, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9083f29b0}) [2025-01-08 00:51:25.094611] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=351] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326285087739, last_flushed_end_lsn:{lsn:43305066}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326285089064, last_flushed_end_lsn:{lsn:43305066}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326285089884, last_flushed_end_lsn:{lsn:43305066}}]) [2025-01-08 00:51:25.094636] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=32780, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326285087739, last_flushed_end_lsn:{lsn:43305066}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326285089064, last_flushed_end_lsn:{lsn:43305066}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326285089884, last_flushed_end_lsn:{lsn:43305066}}]) [2025-01-08 00:51:25.118783] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=102] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3954) [2025-01-08 00:51:25.118836] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:25.120470] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3954) [2025-01-08 00:51:25.120545] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3954) [2025-01-08 00:51:25.122396] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58478][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68B6-0-0] [lt=16] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=128, avg_log_submit_to_flush_time=5552, avg_log_submit_to_slide_time=6625) [2025-01-08 00:51:25.122562] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3954, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:25.163439] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1736326284820404000, v:0}, base_version={val:1736326284759404000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326284962648000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326285163420, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326284820404000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326285117881, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326284820404000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326285220803, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:25.175433] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-01-08 00:51:25.175467] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=27] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:25.186806] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=44] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:25.186841] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.191003] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=70] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=71, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:25.191063] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11630) [2025-01-08 00:51:25.264283] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15132299687, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:25.264351] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:25.264453] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=100] try_wait return eagain(this=0x5572d527f900, thread=0x2ad8bf5f0030) [2025-01-08 00:51:25.264518] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=63] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15132299687, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:25.264555] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=35] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7863188790, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:25.264571] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:25.264579] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try_wait return eagain(this=0x2ad8c0eb0e70, thread=0x2ad9375f57a0) [2025-01-08 00:51:25.264585] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=7] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7863188790, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:25.267824] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:25.271755] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:25.272349] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:25.277936] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-01-08 00:51:25.277980] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=35] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:25.285147] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C0-0-0] [lt=12] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:25.285197] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C0-0-0] [lt=41] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:25.287148] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.302650] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:25.303743] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3907-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=10, avg_log_submit_to_flush_time=3549, avg_log_submit_to_slide_time=31815) [2025-01-08 00:51:25.303875] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=91] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:25.303962] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=79] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17987, miss_cnt:0, cache_read_size:2200352, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:25.314003] INFO [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:197) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] skip repeat study(ret=0, this={this:0x2ad89bf37030, last_study_timestamp:1736325745814828, last_refresh_timestamp:1736326225206761, last_reclaim_timestamp:1736325745818764, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325685391031000, v:0}, is_inited:true}, current_timestamp=1736326285313999, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:51:25.316312] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=46] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1736325745798485000, snapshot_type=1, create_time=1736325745801805, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:25.316391] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1736325745258674000, snapshot_type=1, create_time=1736325745207625, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:25.316411] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1736325685775542000, snapshot_type=1, create_time=1736325685808381, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:25.316421] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1736325745503013000, snapshot_type=2, create_time=1736325745801805, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:25.316431] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1736325744903239000, snapshot_type=2, create_time=1736325745207625, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:25.316441] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1736325685391031000, snapshot_type=2, create_time=1736325685808381, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:25.316450] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1736325745705706000, snapshot_type=3, create_time=1736325745801805, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:25.316459] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1736325745258674000, snapshot_type=3, create_time=1736325745207625, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:25.316468] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1736325685584604000, snapshot_type=3, create_time=1736325685808381, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:25.316477] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325745801805, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:25.316486] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325745207625, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:25.316495] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325685808381, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:25.316937] INFO [MVCC] decide_reserved_snapshot_version_ (ob_multi_version_garbage_collector.cpp:620) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update global reserved snapshot(ret=0, this={this:0x2ad89bf37030, last_study_timestamp:1736325745814828, last_refresh_timestamp:1736326225206761, last_reclaim_timestamp:1736325745818764, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325685391031000, v:0}, is_inited:true}, global_reserved_snapshot_={val:1736325685391031000, v:0}, reserved_snapshot={val:1736325685391031000, v:0}) [2025-01-08 00:51:25.317018] INFO [MVCC] refresh_ (ob_multi_version_garbage_collector.cpp:554) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24] multi-version garbage collector refresh successfully(this={this:0x2ad89bf37030, last_study_timestamp:1736325745814828, last_refresh_timestamp:1736326225206761, last_reclaim_timestamp:1736325745818764, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325685391031000, v:0}, is_inited:true}, collector={reserved_snapshot_version:{val:1736325685391031000, v:0}, reserved_snapshot_type:2, reserved_status:-1, reserved_create_time:1736325685808381, reserved_addr:"192.168.159.139:2882", is_this_server_disabled:false, status:0}) [2025-01-08 00:51:25.317044] INFO [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:232) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=25] repeat refresh successfully(ret=0, this={this:0x2ad89bf37030, last_study_timestamp:1736325745814828, last_refresh_timestamp:1736326285317043, last_reclaim_timestamp:1736325745818764, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325685391031000, v:0}, is_inited:true}, current_timestamp=1736326285314048, GARBAGE_COLLECT_RETRY_INTERVAL=60000000) [2025-01-08 00:51:25.317184] INFO [MVCC] repeat_reclaim (ob_multi_version_garbage_collector.cpp:267) [46354][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=137] skip repeat reclaim(ret=0, this={this:0x2ad89bf37030, last_study_timestamp:1736325745814828, last_refresh_timestamp:1736326285317043, last_reclaim_timestamp:1736325745818764, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325685391031000, v:0}, is_inited:true}, current_timestamp=1736326285317182, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:51:25.319042] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=86, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:25.319124] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=70] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3954, avg_group_log_size=395) [2025-01-08 00:51:25.319492] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.322808] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D42-0-0] [lt=96] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:25.323745] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C20-0-0] [lt=35] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:25.323767] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C20-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6566, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:25.325958] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.326018] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:25.326043] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-01-08 00:51:25.326052] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:25.326064] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:25.326076] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193789233031, flush_io_finished_ret_:0, flush_io_finished_round_:1948, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:25.326271] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=83] new mysql sessid created(conn.sessid_=3221583835, support_ssl=false) [2025-01-08 00:51:25.326416] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221583835, sess.client_addr_="192.168.159.136:39654") [2025-01-08 00:51:25.326461] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=39] accept one succ(*s={this:0x2ad8a0dfc030, session_id:3221583835, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:156}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:25.327442] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=24] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:25.327461] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:292) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=19] mysql login queue(mysql_queue->queue_.size()=1) [2025-01-08 00:51:25.327525] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7E-0-0] [lt=23] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221583835, conn.client_addr_port_=39654, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221583835, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221583835, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D17584A7E-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46333, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523290754803824, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:101, fixup_index:-1, fixup_ash_buffer:{this:0x2ad8c202df98, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:51:25.329111] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7E-0-0] [lt=169][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:51:25.329256] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7E-0-0] [lt=93] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221583835, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:25.329376] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB602-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:25.329469] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB602-0-0] [lt=72] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326284804144004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326284804144004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:25.330988] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:25.331346] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:25.331853] INFO [SERVER] process (obmp_init_db.cpp:53) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AAC-0-0] [lt=50] init db(db_name=oceanbase) [2025-01-08 00:51:25.337723] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1A8BB602-0-0] [lt=20] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.816151428222656250e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.522780101645644208e+02) [2025-01-08 00:51:25.338617] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] Garbage Collector is running(seq_=379, gc_interval=10000000) [2025-01-08 00:51:25.338682] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=41] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:25.338706] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=13] gc_check_member_list_ cost time(ret=0, time_us=47) [2025-01-08 00:51:25.338719] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:51:25.338731] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:25.338743] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:25.338753] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=10) [2025-01-08 00:51:25.338759] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=4] ObSafeDestroyHandler start process [2025-01-08 00:51:25.341134] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:25.341830] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:25.344758] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB602-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1736326285343947000, start_id=1736326285343947000, end_id=1736326285343947001) [2025-01-08 00:51:25.345902] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58503][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:25.348505] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=35] new mysql sessid created(conn.sessid_=3221583837, support_ssl=false) [2025-01-08 00:51:25.348602] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221583837, sess.client_addr_="192.168.159.136:39658") [2025-01-08 00:51:25.348625] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x2ad8efffadb0, session_id:3221583837, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:25.349507] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:25.349585] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7B-0-0] [lt=27] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221583837, conn.client_addr_port_=39658, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221583837, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221583837, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D17784A7B-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46334, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523290814478534, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:76, fixup_index:-1, fixup_ash_buffer:{this:0x2ad8af7cd438, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:51:25.351289] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7B-0-0] [lt=118][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:51:25.351357] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7B-0-0] [lt=17] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221583837, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:25.351638] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=20] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:25.353360] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:25.354421] INFO [SERVER] process (obmp_init_db.cpp:53) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857A6-0-0] [lt=36] init db(db_name=oceanbase) [2025-01-08 00:51:25.358350] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8690, offset=2466912}, end_cursor:ObLogCursor{file_id=1, log_id=8691, offset=2467217}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894abc050, len:2976, log_data_len:305, seq:8690, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:25.358804] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E59-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:25.358852] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E59-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326284800559003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326284800559003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:25.359263] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857A9-0-0] [lt=34] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12358, add_timestamp:1736326285359248}) [2025-01-08 00:51:25.359306] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857A9-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12396, add_timestamp:1736326285359305}) [2025-01-08 00:51:25.362981] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857A9-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:25.363326] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597B-0-0] [lt=30] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12358, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285363302, sql_renew_time:1736326285363302, is_mark_fail:false}]) [2025-01-08 00:51:25.363362] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597B-0-0] [lt=34] success to process renew task(task={tenant_id:1, table_id:12358, add_timestamp:1736326285359248}, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285363302, sql_renew_time:1736326285363302, is_mark_fail:false}]) [2025-01-08 00:51:25.363390] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597C-0-0] [lt=16] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12396, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285363385, sql_renew_time:1736326285363385, is_mark_fail:false}]) [2025-01-08 00:51:25.363406] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597C-0-0] [lt=15] success to process renew task(task={tenant_id:1, table_id:12396, add_timestamp:1736326285359305}, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285363385, sql_renew_time:1736326285363385, is_mark_fail:false}]) [2025-01-08 00:51:25.366402] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1736326285366399, bytes: 876218263, bw: 0.233973 MB/s, add_ts: 1000389, add_bytes: 245434 [2025-01-08 00:51:25.369539] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857AA-0-0] [lt=18] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11001, add_timestamp:1736326285369525}) [2025-01-08 00:51:25.376652] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857AA-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:25.377339] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597D-0-0] [lt=12] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11001, vtable_type={tenant_id:1, type:2}) [2025-01-08 00:51:25.377961] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBB-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:25.377994] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBB-0-0] [lt=30] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:25.378035] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=20] current monitor number(seq_=-1) [2025-01-08 00:51:25.381097] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597D-0-0] [lt=80] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11001, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285381073, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:25.381132] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597D-0-0] [lt=33] success to process renew task(task={tenant_id:1, table_id:11001, add_timestamp:1736326285369525}, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285381073, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:25.381416] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8691, offset=2467217}, end_cursor:ObLogCursor{file_id=1, log_id=8692, offset=2467514}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ac0050, len:2671, log_data_len:297, seq:8691, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:25.382254] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E59-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:25.382313] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E59-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326284800559003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326284800559003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:25.387264] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=299][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.389765] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:25.390757] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58510][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=102] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:25.395564] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=50] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8692, offset=2467514}, end_cursor:ObLogCursor{file_id=1, log_id=8693, offset=2467811}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ac4050, len:2374, log_data_len:297, seq:8692, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:25.396987] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=37] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8693, offset=2467811}, end_cursor:ObLogCursor{file_id=1, log_id=8694, offset=2468108}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ac8050, len:2077, log_data_len:297, seq:8693, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:25.405881] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D876-0-0] [lt=43] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:25.405912] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D876-0-0] [lt=31] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=28815, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:25.407305] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10] ====== tenant freeze timer task ====== [2025-01-08 00:51:25.407333] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71333860, tenant_id=1001) [2025-01-08 00:51:25.407356] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:25.456751] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=110] gc stale ls task succ [2025-01-08 00:51:25.459355] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.459536] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.472106] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=149] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:25.472869] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:25.480802] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=40] ====== tenant freeze timer task ====== [2025-01-08 00:51:25.487685] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.519607] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=36] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.541402] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.543623] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.557270] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=29] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:25.566119] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=149] get number(ret=0, service_type_=0, range=1, base_id=1736326285557723000, start_id=1736326285557723000, end_id=1736326285557723001) [2025-01-08 00:51:25.586985] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84339092}], scn_array=[cnt:1, {val:1736326285557723000, v:0}], curr_size=586, accum_size=2002, time_cost=20502) [2025-01-08 00:51:25.588597] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=77][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.597573] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AC-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:25.597689] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AC-0-0] [lt=54] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326285326952001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326285326952001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:25.598057] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:25.598162] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58505][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=26] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:25.599162] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18689121}], scn_array=[cnt:1, {val:1736326285301506002, v:0}], curr_size=122, accum_size=732, time_cost=10529) [2025-01-08 00:51:25.610746] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=43314153, this={dio_aligned_buf:{buf_write_offset:3638, buf_padding_size:458, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:45795, truncate_used_ts:13431}, log_block_size:67108864, io_fd:{first_id:-1, second_id:160, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=43311104, buf_len=589, total_write_size=87484, total_write_size_after_dio=679936, ob_pwrite_used_ts=1800722, count=145, each_pwrite_cost=12418) [2025-01-08 00:51:25.610788] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43310057}], scn_array=[cnt:1, {val:1736326285589844000, v:0}], curr_size=589, accum_size=6758, time_cost=16897) [2025-01-08 00:51:25.621284] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2200718}], scn_array=[cnt:1, {val:1736326285309520002, v:0}], curr_size=122, accum_size=732, time_cost=10655) [2025-01-08 00:51:25.660885] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=33] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:25.660939] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=44] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:25.660956] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=98) [2025-01-08 00:51:25.669383] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=50] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:25.670144] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58545E-0-0] [lt=15] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:25.670181] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58545E-0-0] [lt=27] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:25.674500] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=50] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1305424, replace_num=15728) [2025-01-08 00:51:25.674532] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:25.675028] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=98] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6731124736, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:25.685864] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:25.685978] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=79] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44891, [delay/us]:prepare:9, schedule:549, submit:983, rt:4383, total:5925) [2025-01-08 00:51:25.685996] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:25.686050] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44892, [delay/us]: prepare:9, schedule:549, submit:983, device:4383, total:5925) [2025-01-08 00:51:25.686064] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:25.686103] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:81635, [delay/us]:prepare:9, schedule:870, submit:889, rt:3147, total:4917) [2025-01-08 00:51:25.686113] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:25.686149] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:81640, [delay/us]: prepare:9, schedule:870, submit:889, device:3147, total:4917) [2025-01-08 00:51:25.686160] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:25.686181] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16328, [delay/us]:prepare:6, schedule:37, submit:1, rt:1542, total:1587) [2025-01-08 00:51:25.686262] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4306, iops:39, bw:167375, [delay/us]:prepare:6, schedule:386, submit:98, rt:2518, total:3009) [2025-01-08 00:51:25.686278] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:25.686310] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16329, [delay/us]: prepare:6, schedule:37, submit:1, device:1542, total:1587) [2025-01-08 00:51:25.686326] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4306, avg_iops:39, avg_bw:167377, [delay/us]: prepare:6, schedule:386, submit:98, device:2518, total:3009) [2025-01-08 00:51:25.686336] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:25.686455] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=106] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4081, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:348, total:356) [2025-01-08 00:51:25.686483] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:25.686587] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=97] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4081, [delay/us]: prepare:0, schedule:0, submit:600000000, device:348, total:356) [2025-01-08 00:51:25.686607] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:25.686624] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:25.686632] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:25.686640] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:25.686649] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=240kB/s, net_out=226kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:25.688855] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.692473] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [46340][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=30] XA scheduler heartbeat task statistics(avg_time=1) [2025-01-08 00:51:25.704983] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:25.706092] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=19] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:25.706116] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=24] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:25.706165] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E9-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:25.706242] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E9-0-0] [lt=73] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:25.729561] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=35] [APPLY STAT CB APPEND COST TIME](cur_stat_count=30, stat_interval=5000000, avg (count/cost)=141, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:25.729587] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=26] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=30, stat_interval=5000000, avg (count/cost)=29089, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:25.729597] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=30, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:25.729604] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=30, stat_interval=5000000, avg (count/cost)=-1354, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:25.729612] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=30, stat_interval=5000000, avg (count/cost)=27876, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:25.729858] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=28] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:25.729896] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:25.729914] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=15] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:25.729928] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:25.733365] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:51:25.734241] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=872] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:25.742344] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597E-0-0] [lt=17] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12274, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:25.742916] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597E-0-0] [lt=21] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12274, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285742893, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285742893, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285742893, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:25.742961] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597E-0-0] [lt=44] success to process renew task(task={tenant_id:1, table_id:12274, add_timestamp:1736326285741649}, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285742893, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285742893, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285742893, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:25.755009] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:25.755050] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=29] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:25.755069] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:25.755077] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:25.755143] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=63] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:25.755157] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=97) [2025-01-08 00:51:25.764519] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] no need to update min/max version(new_version={val:1736326285361604000, v:0}, min_version={val:1736326285361604000, v:0}, max_version={val:1736326285361604000, v:0}) [2025-01-08 00:51:25.772046] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=133] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:25.789482] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=81][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.794382] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=49] ====== check clog disk timer task ====== [2025-01-08 00:51:25.794412] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:25.798318] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:25.800604] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=84] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1002}, {id:1001}]) [2025-01-08 00:51:25.800677] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:25.800708] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=17] gc stale ls task succ [2025-01-08 00:51:25.800763] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:51:25.803773] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:25.803817] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:25.813869] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58526][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D62-0-0] [lt=85] init point thread id with(&point=0x5572d50e50c0, idx_=1182, point=[thread id=58526, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58526) [2025-01-08 00:51:25.814892] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1736326285649980000, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2179856}, palf_committed_end_scn_:{val:1736326285649980001, v:0}, last_check_scn_:{val:1736326285649980001, v:0}, max_applied_cb_scn_:{val:1736326285649980000, v:0}}) [2025-01-08 00:51:25.814930] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=34] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2179856}, readable_log_tail:{lsn:2179856}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64924912, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:25.820006] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185430-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:25.820185] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:25.820209] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:51:25.820323] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=112] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:25.820339] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:25.820352] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193789732208, flush_io_finished_ret_:0, flush_io_finished_round_:1944, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:25.820451] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:25.820470] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-01-08 00:51:25.820477] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:25.820486] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:25.820496] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193789732221, flush_io_finished_ret_:0, flush_io_finished_round_:3772, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:25.821660] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870D0-0-0] [lt=9][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:25.821718] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:25.821735] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:25.823321] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:304K/70:438177 disk:304K/70:438177 t1m2:52K/11:477271 t1m2g0:52K/11:477271 t1002m2:180K/43:359300 t1002m2g0:180K/43:359300 t1001m2:72K/16:619637 t1001m2g0:72K/16:619637 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:25.825783] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3CE-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:25.827143] INFO pn_ratelimit (group.c:653) [58515][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=59] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:25.827627] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=647712768, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=426029056, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:25.827749] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=85] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:25.827769] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=17] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2847, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:25.827831] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1116487680, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2104737792, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:25.827881] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:25.827906] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:25.833093] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D068-0-0] [lt=57][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:25.835567] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=26] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:25.835652] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=77] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:25.835680] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=25] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:25.835710] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=28] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:25.838914] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=29] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:25.840526] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:25.840686] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:25.841050] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=55] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:25.844241] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:25.844426] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:25.845024] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=40] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:25.846085] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:25.846633] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=52] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:25.846673] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=38] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:25.847995] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487B-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:25.848079] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487B-0-0] [lt=38] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:25.848097] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487B-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:25.851438] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=13] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:25.853283] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=49] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:25.854661] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:25.855545] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=38] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:25.855596] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=49] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:25.857099] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487B-0-0] [lt=15] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:25.857197] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487B-0-0] [lt=45] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:25.857291] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487B-0-0] [lt=92] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:25.860411] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=13] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:25.862357] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=39] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:25.865058] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1736326285865055, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008750, add_bytes: 0 [2025-01-08 00:51:25.868984] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=19] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:25.869026] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:25.872846] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487B-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:25.872947] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487B-0-0] [lt=48] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:25.872991] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487B-0-0] [lt=41] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:25.874763] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:25.876567] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:25.873017] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=15] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:25.877514] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=4491] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:25.877597] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=72] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:25.877630] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=31] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:25.877675] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487B-0-0] [lt=39] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:25.877717] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487B-0-0] [lt=41] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:25.877748] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=24] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:25.877774] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=25] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:25.877821] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487B-0-0] [lt=43] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:25.877856] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487B-0-0] [lt=34] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:25.877886] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=24] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:25.877912] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=25] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:25.877950] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487B-0-0] [lt=35] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:25.878002] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487B-0-0] [lt=51] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:25.881254] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=34] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:25.881285] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=28] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:25.881295] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=9] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:25.881334] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=8] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:25.881347] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:25.881356] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:25.881368] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:25.881385] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487B-0-0] [lt=16] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:25.889370] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.896864] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=80] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.896907] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.896933] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.900016] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2179856}], scn_array=[cnt:1, {val:1736326285802951000, v:0}], curr_size=122, accum_size=1098, time_cost=11074) [2025-01-08 00:51:25.900309] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=97][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18690178}], scn_array=[cnt:1, {val:1736326285802951000, v:0}], curr_size=122, accum_size=1789, time_cost=11401) [2025-01-08 00:51:25.902760] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:25.905873] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:195) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA4-0-0] [lt=35][errcode=-4018] table_operator_ get_source failed(ret=-4018) [2025-01-08 00:51:25.907795] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA4-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:25.909610] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA4-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:25.909884] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=322, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:25.909912] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=4, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:25.910647] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3913-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:25.915278] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=55] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2933, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:25.915310] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=540, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:25.915367] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=90] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=46, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:25.917983] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=40] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:25.918951] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF58-0-0] [lt=24] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326286026402, keepalive_ts:-1}) [2025-01-08 00:51:25.918994] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF58-0-0] [lt=41] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326286026402, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:25.931355] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C668-0-0] [lt=23][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193790845186, cache_obj->added_lc()=false, cache_obj->get_object_id()=13634, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:25.935612] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1959C66B-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326285934634000, start_id=1736326285934634000, end_id=1736326285934634001) [2025-01-08 00:51:25.942577] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C66D-0-0] [lt=47] get number(ret=0, service_type_=0, range=1, base_id=1736326285941829000, start_id=1736326285941829000, end_id=1736326285941829001) [2025-01-08 00:51:25.948834] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:25.959308] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1929C9BF-0-0] [lt=104][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193790873139, cache_obj->added_lc()=false, cache_obj->get_object_id()=13635, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:25.963364] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46504][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDF1-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193790877199, cache_obj->added_lc()=false, cache_obj->get_object_id()=13636, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:25.973985] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:25.975670] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58513][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=21] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-01-08 00:51:25.980441] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=195, gc_interval=10000000) [2025-01-08 00:51:25.980489] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=27] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:25.980510] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=12] gc_check_member_list_ cost time(ret=0, time_us=42) [2025-01-08 00:51:25.980523] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:51:25.982242] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:25.982273] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=25] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:25.982286] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] execute_gc cost time(ret=0, time_us=14) [2025-01-08 00:51:25.982295] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:51:25.986249] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326281771630, last_first_register_time_us=1736324454612308, register_time_us=1736326281771630, curr_time_us=1736326285986247) [2025-01-08 00:51:25.986351] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=45] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326285986296, last_submit_register_req_time_us=1736326285986296, register_parent_reason=1) [2025-01-08 00:51:25.986371] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:25.988978] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=66, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:25.989021] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1911, avg_group_log_size=191) [2025-01-08 00:51:25.989040] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1911) [2025-01-08 00:51:25.989054] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:25.989123] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=76, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:25.989144] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:25.989156] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:25.989166] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:25.989681] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:25.991114] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:25.991341] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1911) [2025-01-08 00:51:25.992275] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB APPEND COST TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=101, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:25.992303] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=27] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=40435, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:25.992312] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:25.992320] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=-15463, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:25.992327] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=25074, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:25.993979] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=124] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=10380, avg_group_log_size=648) [2025-01-08 00:51:25.994075] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=89] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:25.996345] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4197, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:25.996387] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10380) [2025-01-08 00:51:25.996408] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=277, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:25.996437] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9913) [2025-01-08 00:51:25.996456] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=26, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:25.997518] INFO [LIB] stat (utility.h:1210) [58522][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D64-0-0] [lt=47] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=31, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:25.997555] INFO [LIB] stat (utility.h:1210) [58522][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D64-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=7169, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:25.997567] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58522][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D64-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=107, avg_log_submit_to_flush_time=4528, avg_log_submit_to_slide_time=7061) [2025-01-08 00:51:25.997688] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10380, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:25.998236] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=56] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326285309520003, v:0}, new_max_version={val:1736326285309520003, v:0}, last_min_version={val:1736326285301506001, v:0}, last_max_version={val:1736326285301506001, v:0}, total_time_us=6293, delta_us=688714) [2025-01-08 00:51:25.998290] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326285309520003, v:0}, min_version={val:1736326285309520003, v:0}, max_version={val:1736326285309520003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:26.002160] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4675-0-0] [lt=20] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:26.002273] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4675-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:26.002292] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4675-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:26.004077] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1975, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:26.004118] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:26.004139] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=95, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:26.004250] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=61, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:26.005043] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3915-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:26.005312] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3915-0-0] [lt=242] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:26.005810] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D881-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=32, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:26.005832] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D881-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13080, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:26.005841] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D881-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=25, avg_log_submit_to_flush_time=2139, avg_log_submit_to_slide_time=13054) [2025-01-08 00:51:26.006104] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:26.006164] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18096, miss_cnt:0, cache_read_size:2213849, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:26.007836] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:26.008641] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58522][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC5-0-0] [lt=11] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326286116344, keepalive_ts:-1}) [2025-01-08 00:51:26.008715] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58522][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC5-0-0] [lt=73] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326286116344, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:26.056821] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dfc030, session_id:3221583835, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326285333539, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:26.056900] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=68] kill and revert session(conn.sessid_=3221583835, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:51:26.056914] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] sock ref clean, do destroy(*s={this:0x2ad8a0dfc030, session_id:3221583835, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326285333539, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:26.056963] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221583835, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:51:26.056976] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221583835, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AAE-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:15063892, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:15063893, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:15063894, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:15063894, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:15063894, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:15063894, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:15063894, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:15063894, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:15063894, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:26.057072] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0de8030, session_id:3221581910, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326269524961, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1496, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:26.057101] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221581910, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:51:26.057109] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sock ref clean, do destroy(*s={this:0x2ad8a0de8030, session_id:3221581910, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326269524961, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1496, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:26.057143] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221581910, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:51:26.057152] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221581910, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B7F-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:18, cur_pkt_pos_:24, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:13682646, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:13682647, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:13682648, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:13682648, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:13682648, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:13682648, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:13682648, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:13}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:13682648, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:14}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:13682648, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:15}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{receive:obp20_header_:{payload_len:194, pkt_seq:0, req_id:13682649, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:190, rec_:194, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[11]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:13682649, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:13682649, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[13]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:13682649, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[14]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:13682649, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[15]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:13682649, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[16]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:13682649, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:13}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[17]:{receive:obp20_header_:{payload_len:410, pkt_seq:0, req_id:13682650, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:406, rec_:410, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[18]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:13682650, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[19]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:13682650, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[20]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:13682650, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[21]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:13682650, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:1, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[22]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:13682650, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[23]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:13682650, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:10}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:26.057487] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=142] free session successfully(sessid=3221583835) [2025-01-08 00:51:26.057509] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] mark session id unused(sessid=3221583835) [2025-01-08 00:51:26.057564] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221581910) [2025-01-08 00:51:26.057575] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221581910) [2025-01-08 00:51:26.067466] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5257493459, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:26.067575] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=38] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:26.067591] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5257493459, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:26.072690] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3926, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:26.072743] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=405, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:26.072838] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:26.075004] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:26.077354] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:26.087649] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BA7-0-0] [lt=12] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:26.090100] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=105][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:26.090931] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4099, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:26.090976] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:26.091001] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=112, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:26.091243] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:26.091426] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5028, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:26.091493] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=66] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2033) [2025-01-08 00:51:26.091509] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=196, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:26.091559] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=32, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:26.092333] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86C7-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:26.092370] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86C7-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5658, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:26.092458] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:26.093159] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86C8-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:26.093239] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86C8-0-0] [lt=78] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6097, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:26.093252] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86C8-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=101, avg_log_submit_to_flush_time=5059, avg_log_submit_to_slide_time=5995) [2025-01-08 00:51:26.093423] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2033, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:26.167314] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=90] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326285941829000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=225458, in_cluster_service=true, cluster_version={val:1736326285755329000, v:0}, min_cluster_version={val:1736326285755329000, v:0}, max_cluster_version={val:1736326285755329000, v:0}, get_cluster_version_err=0, cluster_version_delta=411958, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36570, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=540664274, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:26.186267] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B2BBA1E-0-0] [lt=50] get number(ret=0, service_type_=0, range=1, base_id=1736326286185744000, start_id=1736326286185744000, end_id=1736326286185744001) [2025-01-08 00:51:26.186363] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1003, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18086, max_lsn:2201450, max_scn:{val:1736326285810258002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2201450}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2201328}, last_submit_end_lsn:{lsn:2201450}, last_submit_log_id:18086, last_submit_log_pid:1, max_flushed_lsn:{lsn:2201328}, max_flushed_end_lsn:{lsn:2201450}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2201328}, last_slide_log_id:18085, last_slide_scn:{val:1736326285810258001, v:0}, last_slide_lsn:{lsn:2201206}, last_slide_end_lsn:{lsn:2201328}, last_slide_log_pid:1, last_slide_log_accum_checksum:2871239803, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924fd6530}, [StateMgr]={this:0x2ad924fda830, self:"192.168.159.137:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.139:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18086, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad924fe12f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326285986296, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326285986296, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924fd7c70}, [ModeMgr]={palf_id:1003, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436825152001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/meta", dir_fd:181, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924fdca38, flashback_version:0}, log_storage:{log_tail:{lsn:2201450}, readable_log_tail:{lsn:2201450}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64903318, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, palf_epoch:4, last_purge_throttling_ts:-1, this:0x2ad924fdaf70}, [Reconfirm]={palf_id:1003, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924fda9b0}) [2025-01-08 00:51:26.186658] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=288][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:26.186749] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=88] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:26.186766] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:26.186778] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] current log_task status(palf_id=1003, self="192.168.159.137:2882", log_id=18086, log_task={header:{begin_lsn:{lsn:2201328}, end_lsn:{lsn:2201450}, log_id:18086, min_scn:{val:1736326285810258002, v:0}, max_scn:{val:1736326285810258002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2201206}, prev_proposal_id:1, committed_end_lsn:{lsn:2201328}, data_checksum:940009244, accum_checksum:1144517368, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1736326286108641, freeze_ts:1736326286108641, submit_ts:1736326286108730, flushed_ts:1736326286111023, gen_to_freeze cost time:0, gen_to_submit cost time:89, submit_to_flush cost time:2293}, ack_info_list=[]) [2025-01-08 00:51:26.190111] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:26.190810] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=184, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:26.190849] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=39] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=9263, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:26.190861] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:26.190869] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=-1327, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:26.190876] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=8120, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:26.194370] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get number(ret=0, service_type_=0, range=1, base_id=1736326286193813000, start_id=1736326286193813000, end_id=1736326286193813001) [2025-01-08 00:51:26.194491] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=71, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:26.194517] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9791) [2025-01-08 00:51:26.221430] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5309) [2025-01-08 00:51:26.221536] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=101] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:26.222982] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5309) [2025-01-08 00:51:26.223097] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=161] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=5309) [2025-01-08 00:51:26.225443] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58478][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68C2-0-0] [lt=42] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=116, avg_log_submit_to_flush_time=4398, avg_log_submit_to_slide_time=6930) [2025-01-08 00:51:26.225552] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=83] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5309, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:26.267757] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15137410293, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:26.268177] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=111] recycle one thread(this=0x5572d527f900, total=6, remain=5) [2025-01-08 00:51:26.268373] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=142] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15137410293, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:26.268393] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=19] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7868228518, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:26.268504] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] recycle one thread(this=0x2ad8c0eb0e70, total=6, remain=5) [2025-01-08 00:51:26.268517] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7868228518, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:26.268597] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=89] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:26.275242] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=73] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:26.275856] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:26.277784] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:26.289061] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [50597][T1001_ArcFetche][T1001][YB42C0A89F89-00062B2D1CC8458E-0-0] [lt=47] ObArchiveFetcher is running(thread_index=0) [2025-01-08 00:51:26.290272] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:26.304490] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=53] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=579, get_gts_with_stc_cnt=30, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=28, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:26.304650] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=157] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:26.304683] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=491, get_gts_with_stc_cnt=69, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=66, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:26.309366] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D888-0-0] [lt=29] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=43, avg_log_submit_to_flush_time=3372, avg_log_submit_to_slide_time=20977) [2025-01-08 00:51:26.309518] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:26.309554] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17998, miss_cnt:0, cache_read_size:2201694, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:26.310107] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50915][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE89B-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1736326286308837000, start_id=1736326286308837000, end_id=1736326286308837001) [2025-01-08 00:51:26.318000] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D170BBDB4-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1736326286317586000, start_id=1736326286317586000, end_id=1736326286317586001) [2025-01-08 00:51:26.319764] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=58] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.320104] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.322387] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D250BAD50-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326286319998000, start_id=1736326286319998000, end_id=1736326286319998001) [2025-01-08 00:51:26.324010] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C2C-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=47, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:26.324034] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C2C-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=7193, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:26.326285] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=53] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:26.326374] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=81] temporary file module use no blocks [2025-01-08 00:51:26.326387] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:26.326400] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:26.326508] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=107] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193790239974, flush_io_finished_ret_:0, flush_io_finished_round_:1949, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:26.340653] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:26.340776] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:26.344329] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=95] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:26.344680] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=66] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-01-08 00:51:26.346238] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=48] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:26.354977] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=92] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:26.369483] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=42] PNIO [ratelimit] time: 1736326286369478, bytes: 876456494, bw: 0.226497 MB/s, add_ts: 1003079, add_bytes: 238231 [2025-01-08 00:51:26.369695] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB603-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:26.369746] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB603-0-0] [lt=34] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326285810258004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326285810258004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:26.381041] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBC-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:26.381109] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBC-0-0] [lt=62] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:26.390473] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:26.391263] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=17] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:26.393915] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:26.394151] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58512][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=8] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:26.397719] INFO [STORAGE] runTimerTask (ob_ddl_heart_beat_task.cpp:49) [58515][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=90] send to rs all task status succeed [2025-01-08 00:51:26.406984] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=51] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8694, offset=2468108}, end_cursor:ObLogCursor{file_id=1, log_id=8695, offset=2468413}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894acc050, len:1780, log_data_len:305, seq:8694, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:26.408031] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5A-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:26.408092] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5A-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326286304762000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326286304762000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:26.412747] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E5A-0-0] [lt=85] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.840151977539062500e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=392, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.431660989222258422e+02) [2025-01-08 00:51:26.422372] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=112] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8695, offset=2468413}, end_cursor:ObLogCursor{file_id=1, log_id=8696, offset=2468710}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ad0050, len:1475, log_data_len:297, seq:8695, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:26.422914] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=75, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:26.422951] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5854, avg_group_log_size=450) [2025-01-08 00:51:26.423523] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5A-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:26.423572] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5A-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326286304762000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326286304762000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:26.436409] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=62] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8696, offset=2468710}, end_cursor:ObLogCursor{file_id=1, log_id=8697, offset=2469007}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ad4050, len:1178, log_data_len:297, seq:8696, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:26.438406] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=114] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8697, offset=2469007}, end_cursor:ObLogCursor{file_id=1, log_id=8698, offset=2469304}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ad8050, len:881, log_data_len:297, seq:8697, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:26.456862] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=47] gc stale ls task succ [2025-01-08 00:51:26.459232] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.459578] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.475466] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:26.478333] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:26.478580] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58530][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:26.478805] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:51:26.490643] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:26.504944] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5990) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=42] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000) [2025-01-08 00:51:26.504984] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505053] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505117] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505274] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=86] get multi version start(ls_id={id:1}, tablet_id={id:60259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.505994] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506052] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506173] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506326] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=98] get multi version start(ls_id={id:1}, tablet_id={id:50149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506445] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506504] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101098}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506871] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.506964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507032] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101036}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507158] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507418] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101064}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507542] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101080}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:50424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.507937] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508168] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=58] get multi version start(ls_id={id:1}, tablet_id={id:50437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508315] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=88] get multi version start(ls_id={id:1}, tablet_id={id:101083}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508604] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508663] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101018}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508722] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508900] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.508958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101034}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509285] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101084}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509436] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509497] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509562] INFO [LIB] stat (utility.h:1210) [58525][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC391F-0-0] [lt=43] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:26.509588] INFO [LIB] stat (utility.h:1210) [58525][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC391F-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=32915, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:26.509616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509599] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58525][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC391F-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18089, log_task={header:{begin_lsn:{lsn:2201694}, end_lsn:{lsn:2201816}, log_id:18089, min_scn:{val:1736326286308837000, v:0}, max_scn:{val:1736326286308837000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2201572}, prev_proposal_id:1, committed_end_lsn:{lsn:2201694}, data_checksum:940009244, accum_checksum:1661265025, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326286405247, freeze_ts:1736326286405247, submit_ts:1736326286405254, flushed_ts:1736326286407960, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:2706}, fs_cb_begin_ts=1736326286509552, log_life_time=104305) [2025-01-08 00:51:26.509676] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101068}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509910] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.509968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510026] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510176] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510331] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510397] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510684] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101073}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510915] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.510973] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101099}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101092}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511174] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511316] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511442] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511591] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101031}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511829] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511900] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.511971] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:101111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512030] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512146] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512254] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512318] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512377] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512493] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101067}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512610] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512669] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512757] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:101027}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512880] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512938] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.512996] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.513247] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.513313] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.513440] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=75] get multi version start(ls_id={id:1}, tablet_id={id:60465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.513508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.513628] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.513743] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.513803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.513908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=52] get multi version start(ls_id={id:1}, tablet_id={id:101082}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.513967] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514156] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=55] get multi version start(ls_id={id:1}, tablet_id={id:60104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514330] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=30] get multi version start(ls_id={id:1}, tablet_id={id:60245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.514947] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.515009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101057}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.515126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.517357] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2214337}], scn_array=[cnt:1, {val:1736326286319998001, v:0}], curr_size=122, accum_size=610, time_cost=13614) [2025-01-08 00:51:26.517717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.517867] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.517939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518002] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518110] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518230] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101033}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518297] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101066}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=73] get multi version start(ls_id={id:1}, tablet_id={id:50305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518774] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.518893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519124] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519236] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101030}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:101020}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519534] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101044}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519594] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519653] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519761] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519826] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.519945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520121] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520376] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520466] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520537] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101013}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520664] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520919] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101072}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.520980] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521040] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101088}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521100] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521160] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101069}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521220] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521466] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521509] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=72] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.521586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:50245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521778] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101060}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521896] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.521953] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101100}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522011] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522261] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101022}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:60515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522819] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522930] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.522993] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523052] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101035}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523337] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:5}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:50145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523735] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101058}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101081}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.523967] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101086}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101052}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524141] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524250] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101019}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524369] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101079}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524566] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=30] get multi version start(ls_id={id:1}, tablet_id={id:60306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524764] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524823] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:1}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.524998] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.525056] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.525352] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=105] get multi version start(ls_id={id:1}, tablet_id={id:50377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.525428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.525600] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.525670] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.525734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.525796] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.525928] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.526144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=82] get multi version start(ls_id={id:1}, tablet_id={id:50436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.526311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.526392] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.526476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.526559] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.527184] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43319805}], scn_array=[cnt:1, {val:1736326286502031000, v:0}], curr_size=589, accum_size=6126, time_cost=23916) [2025-01-08 00:51:26.530098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530247] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:50438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530322] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530450] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530512] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530574] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530636] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530697] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530852] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530914] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.530975] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.531036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.531097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.531158] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.531218] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.531639] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.531742] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:60003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.531828] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.531893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.531954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532016] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532198] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532394] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532518] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532603] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532861] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532922] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.532982] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533104] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533171] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101043}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533302] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:50378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533366] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533460] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533646] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533769] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533829] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.533951] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534270] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101032}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534339] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534454] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534590] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534650] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534833] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.534954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535014] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535176] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535238] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535368] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535492] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101028}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535553] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101021}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.535936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536066] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536187] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536247] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536376] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=71] get multi version start(ls_id={id:1}, tablet_id={id:50498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536438] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536499] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536594] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.536945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537259] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537333] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537395] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101037}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537517] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537577] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537755] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101054}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.537943] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101041}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538187] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101097}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538322] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=75] get multi version start(ls_id={id:1}, tablet_id={id:60210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538386] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101024}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538630] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538691] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.538913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539117] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:50190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539261] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=86] get multi version start(ls_id={id:1}, tablet_id={id:60423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539448] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101055}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:50496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539676] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539799] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539920] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.539981] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540102] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540162] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101063}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540280] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540411] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:3}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540614] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540691] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:101040}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540753] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.540996] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.541057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101047}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.541118] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.541178] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.541323] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:50514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.541401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.542561] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.542638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.542700] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101059}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.542761] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.542835] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.542911] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.542988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543247] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543321] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543384] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543445] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543505] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543566] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543627] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543748] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543808] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.543893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544038] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=84] get multi version start(ls_id={id:1}, tablet_id={id:60413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544102] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544287] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544353] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544414] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544536] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544807] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544880] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.544943] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101046}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545186] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545664] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545912] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.545973] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546033] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546094] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101045}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546279] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546359] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546439] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546564] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546625] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546685] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:60499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.546955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101078}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547156] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547236] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101049}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547488] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547624] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:50209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547683] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101101}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.547990] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548051] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548214] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:50330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548424] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101091}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548485] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548560] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548619] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101039}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548735] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101029}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548868] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.548990] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549289] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549417] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549477] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.549730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.592287] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2201816}], scn_array=[cnt:1, {val:1736326286308837001, v:0}], curr_size=122, accum_size=610, time_cost=82560) [2025-01-08 00:51:26.592566] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4015] clock out of order(cur_ts=1736326286592563, cur_ts_=1736326286531320, delta=61243) [2025-01-08 00:51:26.593883] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.593922] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.600903] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:26.601097] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=15] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:26.603300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.603456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:50206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.603525] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.603589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.603651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.603712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.603772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.603831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.603959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.604024] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101095}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.604084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101070}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.604143] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.604282] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.604442] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.604620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:101075}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.604800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.604949] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:60373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:60197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605194] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101077}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605332] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605457] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101071}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605722] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101065}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605902] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.605986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.606306] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=157] get multi version start(ls_id={id:1}, tablet_id={id:214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.606415] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BA9-0-0] [lt=30] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:26.606451] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.606549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.606614] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:100003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.606676] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.606737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.606850] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.606915] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.606991] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101014}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607259] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607325] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607560] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607619] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607736] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607892] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:60394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.607955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608014] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608022] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:26.608073] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608132] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608208] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58522][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=17] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:51:26.608292] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608425] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608564] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608636] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608755] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608813] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608930] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.608988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101050}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609106] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609229] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609298] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609417] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609497] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609627] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101056}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609745] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609922] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.609981] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610040] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101023}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:101007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610384] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101048}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101026}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610561] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610795] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610912] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.610970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.611029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.611088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.611164] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.611432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.611506] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:101025}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.611720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.611789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.611943] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:50196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:60132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612043] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58525][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3921-0-0] [lt=33][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18090, log_task={header:{begin_lsn:{lsn:2201816}, end_lsn:{lsn:2201938}, log_id:18090, min_scn:{val:1736326286308837001, v:0}, max_scn:{val:1736326286308837001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2201694}, prev_proposal_id:1, committed_end_lsn:{lsn:2201816}, data_checksum:940009244, accum_checksum:3600639022, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326286509533, freeze_ts:1736326286509533, submit_ts:1736326286509541, flushed_ts:1736326286597342, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:87801}, fs_cb_begin_ts=1736326286612019, log_life_time=102486) [2025-01-08 00:51:26.612125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612274] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:4}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:100005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612446] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612572] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612634] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612814] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:60512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.612943] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.613059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.613128] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616292] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:50254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616546] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101085}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616735] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.616957] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617145] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101087}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617269] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617400] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617461] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617663] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=81] get multi version start(ls_id={id:1}, tablet_id={id:50469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617905] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.617969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101051}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618093] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618185] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=81] get multi version start(ls_id={id:1}, tablet_id={id:60237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618388] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101061}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618450] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618512] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618514] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB02A-0-0] [lt=113] get number(ret=0, service_type_=0, range=1, base_id=1736326286614894000, start_id=1736326286614894000, end_id=1736326286614894001) [2025-01-08 00:51:26.618573] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618635] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618699] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101076}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618821] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.618996] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619123] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619185] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619403] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=115] get multi version start(ls_id={id:1}, tablet_id={id:60114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619474] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619629] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619697] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:50247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101074}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.619935] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620002] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620123] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101053}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620184] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620297] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620363] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620424] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620605] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.620924] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:60474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621261] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621448] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621629] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621689] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621749] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621843] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:60241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.621969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622030] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622273] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101042}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622398] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622459] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622518] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622579] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622639] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622732] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622813] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101096}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.622997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.623057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.623147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.623215] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.623384] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.623452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.623540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.623631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.623827] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=86] get multi version start(ls_id={id:1}, tablet_id={id:60222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.624023] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.624295] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=75] get multi version start(ls_id={id:1}, tablet_id={id:101038}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.624420] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.624582] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.624649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.624711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.624831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.624897] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101090}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.625147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.625326] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.625406] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.625559] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=97] get multi version start(ls_id={id:1}, tablet_id={id:373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324475608151000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324475608151000, max_merged_snapshot=1) [2025-01-08 00:51:26.626715] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18691455}], scn_array=[cnt:1, {val:1736326286304762002, v:0}], curr_size=122, accum_size=1033, time_cost=29534) [2025-01-08 00:51:26.627667] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=184][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2201938}], scn_array=[cnt:1, {val:1736326286308837002, v:0}], curr_size=122, accum_size=732, time_cost=14543) [2025-01-08 00:51:26.628028] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:116) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853C2-0-0] [lt=15] SSTableGCTask(cost_ts=123515) [2025-01-08 00:51:26.632107] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2214459}], scn_array=[cnt:1, {val:1736326286319998002, v:0}], curr_size=122, accum_size=732, time_cost=29258) [2025-01-08 00:51:26.642316] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB19B-0-0] [lt=68] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:26.642427] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB19B-0-0] [lt=102] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:26.642463] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB19B-0-0] [lt=32] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:26.643935] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AD-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:26.643976] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AD-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326286319998001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326286319998001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:26.650867] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1EBBB19D-0-0] [lt=31] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326286647332, can_tenant_elr:true}) [2025-01-08 00:51:26.677141] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:26.680778] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=88] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:26.688108] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BECD-0-0] [lt=48] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:26.695319] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:26.695448] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:12, bw:56807, [delay/us]:prepare:19, schedule:222, submit:97, rt:1935, total:2274) [2025-01-08 00:51:26.695469] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:26.695772] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=283] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:12, avg_bw:56792, [delay/us]: prepare:19, schedule:222, submit:97, device:1935, total:2274) [2025-01-08 00:51:26.695842] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=62] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:26.695887] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:15, bw:68957, [delay/us]:prepare:8, schedule:201, submit:154, rt:2697, total:3060) [2025-01-08 00:51:26.695899] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:26.696012] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:15, avg_bw:68956, [delay/us]: prepare:8, schedule:201, submit:154, device:2697, total:3060) [2025-01-08 00:51:26.696029] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:26.696129] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=71] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16222, [delay/us]:prepare:6, schedule:62, submit:1, rt:1315, total:1385) [2025-01-08 00:51:26.696163] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:39, bw:158172, [delay/us]:prepare:9, schedule:572, submit:908, rt:3428, total:4919) [2025-01-08 00:51:26.696178] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:26.696297] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=112] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16222, [delay/us]: prepare:6, schedule:62, submit:1, device:1315, total:1385) [2025-01-08 00:51:26.696315] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:39, avg_bw:158164, [delay/us]: prepare:9, schedule:572, submit:908, device:3428, total:4919) [2025-01-08 00:51:26.696325] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:26.696347] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4055, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:398, total:408) [2025-01-08 00:51:26.696385] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:26.696444] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=54] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4056, [delay/us]: prepare:0, schedule:0, submit:600000000, device:398, total:408) [2025-01-08 00:51:26.696463] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:26.696480] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:26.696488] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:26.696496] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:26.696504] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=294kB/s, net_out=294kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:26.701014] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:26.703460] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4683-0-0] [lt=93][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18194, log_task={header:{begin_lsn:{lsn:2214459}, end_lsn:{lsn:2214581}, log_id:18194, min_scn:{val:1736326286319998002, v:0}, max_scn:{val:1736326286319998002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2214337}, prev_proposal_id:1, committed_end_lsn:{lsn:2214459}, data_checksum:1715074492, accum_checksum:949788432, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326286602753, freeze_ts:1736326286602753, submit_ts:1736326286602760, flushed_ts:1736326286632178, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:29418}, fs_cb_begin_ts=1736326286703426, log_life_time=100673) [2025-01-08 00:51:26.720374] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=27] ====== tenant freeze timer task ====== [2025-01-08 00:51:26.731013] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58530][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C30-0-0] [lt=27] init point thread id with(&point=0x5572d50e51c0, idx_=1186, point=[thread id=58530, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58530) [2025-01-08 00:51:26.761328] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=24] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:26.762738] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=98] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:26.762907] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=95] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326284334180000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:26.762958] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=48] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326284334180000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326284334180000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:26.788446] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:26.789969] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=71] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:26.790154] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=110] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326285633619000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:26.790261] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=103] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326285633619000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326285633619000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:26.801133] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:26.801259] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1002}, {id:1003}]) [2025-01-08 00:51:26.801453] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:26.801751] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:26.802089] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:51:26.806491] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=216][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:26.806520] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:26.806613] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:26.812411] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=84][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84345531}], scn_array=[cnt:1, {val:1736326286797881000, v:0}], curr_size=586, accum_size=3132, time_cost=11319) [2025-01-08 00:51:26.816934] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=158] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:26.819207] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870D6-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:26.819340] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=91][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:26.819353] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:26.822352] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185431-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:26.822617] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=83] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:26.822641] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:51:26.822649] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:26.822662] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:26.822674] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193790734281, flush_io_finished_ret_:0, flush_io_finished_round_:1945, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:26.822714] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:26.822729] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-01-08 00:51:26.822733] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=4] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:26.822740] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:26.822749] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193790734360, flush_io_finished_ret_:0, flush_io_finished_round_:3773, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:26.829632] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=62] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:26.830973] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=24] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:26.831064] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326285742394000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:26.834052] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=24] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:26.834077] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=20] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:26.834139] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=59] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=191) [2025-01-08 00:51:26.834151] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=47] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:26.834181] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=191) [2025-01-08 00:51:26.839346] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3D1-0-0] [lt=28][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:26.840658] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D06B-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:26.841307] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=51] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:26.841259] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:26.842779] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=67] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:26.846155] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:26.846304] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=19] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:26.847810] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=48] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:26.855350] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=78] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:26.871807] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=85] PNIO [ratelimit] time: 1736326286871802, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006747, add_bytes: 0 [2025-01-08 00:51:26.875244] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50503][T1002_L0_G0][T1002][YB42C0A89F8A-00062B2D1B49C757-0-0] [lt=58] get number(ret=0, service_type_=0, range=1, base_id=1736326286874645000, start_id=1736326286874645000, end_id=1736326286874645001) [2025-01-08 00:51:26.875822] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=34] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:300K/72:553610 disk:300K/72:553610 t1m2:52K/12:815382 t1m2g0:52K/12:815382 t1002m2:180K/45:505911 t1002m2g0:180K/45:505911 t1001m2:68K/15:486786 t1001m2g0:68K/15:486786 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:26.877390] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=106] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:26.881385] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:26.883286] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=54] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=912253, clean_num=31457) [2025-01-08 00:51:26.888411] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDE-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:26.888455] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDE-0-0] [lt=37] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=47) [2025-01-08 00:51:26.896531] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.896637] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.896676] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.896702] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:26.898916] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get_min_unreplayed_log_info(lsn={lsn:2214703}, scn={val:1736326286319998004, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326286711315, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2214703}, next_to_submit_scn_:{val:1736326286319998004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2214581}, end_lsn:{lsn:2214703}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2214825}, readable_log_tail:{lsn:2214825}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64889943, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326286319998003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326286319998003, v:0}, data_checksum:3348837161, flag:8296}}, init_mode_version:0, accumulate_checksum:2863319423, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326286319998003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18103, miss_cnt:0, cache_read_size:2214703, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:26.901946] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [46323][T1_RFLWorker][T1][YB42C0A89F89-00062B2D1618458E-0-0] [lt=48] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:51:26.905552] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=106] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:51:26.905606] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=52] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=6) [2025-01-08 00:51:26.905619] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:51:26.905646] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:51:26.907318] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:26.922197] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA5-0-0] [lt=41][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:26.923976] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA5-0-0] [lt=110][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:26.968141] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856E8-0-0] [lt=41] InfoPoolResizeTask(cost_ts=11) [2025-01-08 00:51:26.997673] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=88, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:26.997731] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=2066, avg_group_log_size=206) [2025-01-08 00:51:26.997753] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2066) [2025-01-08 00:51:26.997769] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:26.997872] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:26.997904] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:26.997919] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:26.997976] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:26.999050] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=206] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326286304762003, v:0}, base_version={val:1736326286304762001, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326286304762004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326286999035, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326286304762003, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326286928817, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326286304762003, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326286934279, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:27.001494] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9821, avg_group_log_size=654) [2025-01-08 00:51:27.001520] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:27.002473] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86E1-0-0] [lt=99] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:27.003421] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC86E2-0-0] [lt=81] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:51:27.004118] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=122] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=4347, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:27.004146] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9821) [2025-01-08 00:51:27.004166] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=432, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:27.004261] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9821) [2025-01-08 00:51:27.004284] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=16, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:27.005930] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=86] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326286531320001, v:0}, new_max_version={val:1736326286531320001, v:0}, last_min_version={val:1736326286468427000, v:0}, last_max_version={val:1736326286468427000, v:0}, total_time_us=9475, delta_us=474607) [2025-01-08 00:51:27.005966] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326286531320001, v:0}, min_version={val:1736326286531320001, v:0}, max_version={val:1736326286531320001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:27.007554] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.008018] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3927-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:27.008064] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3927-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:27.008081] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3927-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:27.008101] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3927-0-0] [lt=8][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18093, log_task={header:{begin_lsn:{lsn:2202182}, end_lsn:{lsn:2202304}, log_id:18093, min_scn:{val:1736326286778356000, v:0}, max_scn:{val:1736326286778356000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2202060}, prev_proposal_id:1, committed_end_lsn:{lsn:2202182}, data_checksum:940009244, accum_checksum:640896416, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326286810928, freeze_ts:1736326286810928, submit_ts:1736326286810938, flushed_ts:1736326286818548, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:7610}, fs_cb_begin_ts=1736326287008093, log_life_time=197165) [2025-01-08 00:51:27.008446] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=100, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:27.008541] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=93] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=8, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:27.009656] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=293] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12052, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:27.009681] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:27.009712] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=292, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:27.009875] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=34, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:27.012637] INFO [LIB] stat (utility.h:1210) [58523][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D73-0-0] [lt=55] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:27.012660] INFO [LIB] stat (utility.h:1210) [58523][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D73-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=8402, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:27.012670] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58523][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2D73-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=107, avg_log_submit_to_flush_time=4994, avg_log_submit_to_slide_time=8295) [2025-01-08 00:51:27.012917] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9821, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:27.013125] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get_max_applied_scn(scn={val:1736326286874645000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18692488}, palf_committed_end_scn_:{val:1736326286874645001, v:0}, last_check_scn_:{val:1736326286874645001, v:0}, max_applied_cb_scn_:{val:1736326286874645000, v:0}}) [2025-01-08 00:51:27.013161] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18692488}, readable_log_tail:{lsn:18692488}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48412280, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:27.013320] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=133] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:27.016958] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4687-0-0] [lt=231] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:27.017008] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4687-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:27.017026] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4687-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:27.017047] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4687-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=22, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:27.017057] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4687-0-0] [lt=9] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=64570, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:27.017126] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4687-0-0] [lt=68] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=8, avg_log_submit_to_flush_time=7490, avg_log_submit_to_slide_time=64562) [2025-01-08 00:51:27.017157] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4687-0-0] [lt=29][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18196, log_task={header:{begin_lsn:{lsn:2214703}, end_lsn:{lsn:2214825}, log_id:18196, min_scn:{val:1736326286767852000, v:0}, max_scn:{val:1736326286767852000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2214581}, prev_proposal_id:1, committed_end_lsn:{lsn:2214703}, data_checksum:1715074492, accum_checksum:736222535, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326286815408, freeze_ts:1736326286815408, submit_ts:1736326286815416, flushed_ts:1736326286819139, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:3723}, fs_cb_begin_ts=1736326287017038, log_life_time=201630) [2025-01-08 00:51:27.017417] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=976, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:27.017545] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=120] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18104, miss_cnt:0, cache_read_size:2214825, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:27.019105] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=31] LogLoopThread round_cost_time(us)(round_cost_time=16) [2025-01-08 00:51:27.019410] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=6743, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:27.019460] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:27.019482] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=152, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:27.019618] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:27.028857] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C66F-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193791942690, cache_obj->added_lc()=false, cache_obj->get_object_id()=13637, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:27.029726] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=132] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:27.069853] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5259705271, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:27.069905] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:27.069918] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad908547e00) [2025-01-08 00:51:27.069927] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5259705271, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:27.080801] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=82] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:27.080896] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=78] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=105) [2025-01-08 00:51:27.095489] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=5] Garbage Collector is running(seq_=196, gc_interval=10000000) [2025-01-08 00:51:27.095542] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=29] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:27.095558] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=13] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:27.101779] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84347248}, readable_log_tail:{lsn:84347248}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49862288, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:27.101979] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=197] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:27.102015] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:27.104449] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:27.104477] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6238, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:27.104524] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58515][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=95] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:27.104538] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=887, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:27.104625] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] clock out of order(cur_ts=1736326287104623, cur_ts_=1736326287044195, delta=60428) [2025-01-08 00:51:27.104687] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:27.104703] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9174, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:27.104733] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=36, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:27.105131] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:51:27.107863] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46379][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD7-0-0] [lt=36] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10006, add_timestamp:1736326287107790}) [2025-01-08 00:51:27.108536] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] get number(ret=0, service_type_=0, range=1, base_id=1736326287044195000, start_id=1736326287044195000, end_id=1736326287044195001) [2025-01-08 00:51:27.108579] WDIAG [SHARE] inner_to_show_str (ob_system_variable.cpp:516) [46379][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD7-0-0] [lt=28][errcode=0] sys var casted obj is not ObVarcharType(ret=0, value={"NULL":"NULL"}, *res_obj={"NULL":"NULL"}, get_name()=secure_file_priv) [2025-01-08 00:51:27.108785] WDIAG [SHARE] inner_to_show_str (ob_system_variable.cpp:516) [46379][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CD7-0-0] [lt=20][errcode=0] sys var casted obj is not ObVarcharType(ret=0, value={"NULL":"NULL"}, *res_obj={"NULL":"NULL"}, get_name()=external_user) [2025-01-08 00:51:27.109261] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597F-0-0] [lt=26] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10006, locations=[{table_id:10006, partition_id:10006, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287109184, sql_renew_time:1736326287109184, is_mark_fail:false}]) [2025-01-08 00:51:27.109308] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28597F-0-0] [lt=46] success to process renew task(task={tenant_id:1, table_id:10006, add_timestamp:1736326287107790}, locations=[{table_id:10006, partition_id:10006, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287109184, sql_renew_time:1736326287109184, is_mark_fail:false}]) [2025-01-08 00:51:27.109770] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.111543] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=22] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:51:27.111572] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=22] gc_check_member_list_ cost time(ret=0, time_us=16053) [2025-01-08 00:51:27.111589] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:27.113546] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:27.115133] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=20] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:27.116649] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:27.117684] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB95-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193792031519, cache_obj->added_lc()=false, cache_obj->get_object_id()=13638, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:27.118746] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:27.118788] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=33] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:27.118810] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=21] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:27.118827] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=16] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:27.118838] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:27.118861] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=21] execute_gc cost time(ret=0, time_us=74) [2025-01-08 00:51:27.118870] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyHandler start process [2025-01-08 00:51:27.119872] INFO [STORAGE.BLKMGR] mark_macro_blocks (ob_block_manager.cpp:1076) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=50] no block alloc/free, no need to mark blocks(ret=0, mark_info.count()=0) [2025-01-08 00:51:27.119895] INFO [STORAGE.BLKMGR] mark_and_sweep (ob_block_manager.cpp:1056) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=20] finish once mark and sweep(ret=0, alloc_num=2962, mark_info.count()=0, marker_status={total_block_count:1024, reserved_block_count:2, linked_block_count:0, tmp_file_count:0, data_block_count:11, shared_data_block_count:11, index_block_count:0, ids_block_count:0, disk_block_count:0, bloomfiter_count:0, hold_count:41, pending_free_count:1, free_count:981, shared_meta_block_count:19, mark_cost_time:77, sweep_cost_time:0, start_time:"2025-01-08 00:51:27.119816", last_end_time:"2025-01-08 00:20:57.6873", hold_info:nothing, mark_finished:false}, map_cnt=41) [2025-01-08 00:51:27.120796] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2202426}], scn_array=[cnt:1, {val:1736326286813632001, v:0}], curr_size=122, accum_size=122, time_cost=11646) [2025-01-08 00:51:27.121019] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5385, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:27.121043] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2181320}], scn_array=[cnt:1, {val:1736326286874645002, v:0}], curr_size=122, accum_size=1220, time_cost=17024) [2025-01-08 00:51:27.121060] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:27.121075] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1115, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:27.121130] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:27.121148] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8901, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:27.121167] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=6] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:27.121278] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:27.122318] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58525][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4689-0-0] [lt=39][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18197, log_task={header:{begin_lsn:{lsn:2214825}, end_lsn:{lsn:2214947}, log_id:18197, min_scn:{val:1736326286833111000, v:0}, max_scn:{val:1736326286833111000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2214703}, prev_proposal_id:1, committed_end_lsn:{lsn:2214825}, data_checksum:1715074492, accum_checksum:3575298073, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326287016947, freeze_ts:1736326287016947, submit_ts:1736326287017001, flushed_ts:1736326287019541, gen_to_freeze cost time:0, gen_to_submit cost time:54, submit_to_flush cost time:2540}, fs_cb_begin_ts=1736326287122307, log_life_time=105360) [2025-01-08 00:51:27.125421] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43324089}], scn_array=[cnt:1, {val:1736326287044195000, v:0}], curr_size=589, accum_size=589, time_cost=16687) [2025-01-08 00:51:27.134906] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58533][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:27.135006] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58533][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A0C-0-0] [lt=68] init point thread id with(&point=0x5572d50e5280, idx_=1189, point=[thread id=58533, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58533) [2025-01-08 00:51:27.135310] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:51:27.139794] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=160] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2826, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:27.139831] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1032, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:27.139908] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:27.147095] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221584065, support_ssl=false) [2025-01-08 00:51:27.147158] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] sm conn init succ(conn.sessid_=3221584065, sess.client_addr_="127.0.0.1:51662") [2025-01-08 00:51:27.147176] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x2ad8efff8030, session_id:3221584065, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:156}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.147271] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221584066, support_ssl=false) [2025-01-08 00:51:27.147313] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221584066, sess.client_addr_="127.0.0.1:51666") [2025-01-08 00:51:27.147326] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8efff8db0, session_id:3221584066, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:183}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.147351] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221584067, support_ssl=false) [2025-01-08 00:51:27.147381] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221584067, sess.client_addr_="127.0.0.1:51670") [2025-01-08 00:51:27.147393] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efffc030, session_id:3221584067, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:193}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.147416] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221584068, support_ssl=false) [2025-01-08 00:51:27.147444] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221584068, sess.client_addr_="127.0.0.1:51678") [2025-01-08 00:51:27.147456] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efffcdb0, session_id:3221584068, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:194}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.147494] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221584069, support_ssl=false) [2025-01-08 00:51:27.147543] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221584069, sess.client_addr_="127.0.0.1:51680") [2025-01-08 00:51:27.147561] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2ad8efe0a030, session_id:3221584069, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:195}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.147648] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=44] new mysql sessid created(conn.sessid_=3221584071, support_ssl=false) [2025-01-08 00:51:27.147689] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221584071, sess.client_addr_="127.0.0.1:51668") [2025-01-08 00:51:27.147702] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8efe0adb0, session_id:3221584071, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:196}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.147725] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221584070, support_ssl=false) [2025-01-08 00:51:27.147754] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221584070, sess.client_addr_="127.0.0.1:51674") [2025-01-08 00:51:27.147766] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efe0c030, session_id:3221584070, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:197}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.148054] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CDB-0-0] [lt=5] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12276, add_timestamp:1736326287148036}) [2025-01-08 00:51:27.148675] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285980-0-0] [lt=22] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12276, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:27.149512] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.149563] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=22] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.149581] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.149602] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.149767] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A68-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.150276] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=127] new mysql sessid created(conn.sessid_=3221584072, support_ssl=false) [2025-01-08 00:51:27.150353] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221584072, sess.client_addr_="127.0.0.1:51672") [2025-01-08 00:51:27.150373] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x2ad8a0dfc030, session_id:3221584072, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:198}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.150405] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221584073, support_ssl=false) [2025-01-08 00:51:27.150528] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221584073, sess.client_addr_="127.0.0.1:51676") [2025-01-08 00:51:27.150546] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2ad8951f6030, session_id:3221584073, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:200}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.150572] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221584074, support_ssl=false) [2025-01-08 00:51:27.150605] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221584074, sess.client_addr_="127.0.0.1:51684") [2025-01-08 00:51:27.150618] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8951f6db0, session_id:3221584074, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:201}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.150642] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221584075, support_ssl=false) [2025-01-08 00:51:27.150671] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221584075, sess.client_addr_="127.0.0.1:51686") [2025-01-08 00:51:27.150683] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8951f8030, session_id:3221584075, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:202}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.150734] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=69] new mysql sessid created(conn.sessid_=3221584076, support_ssl=false) [2025-01-08 00:51:27.150774] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221584076, sess.client_addr_="127.0.0.1:51664") [2025-01-08 00:51:27.150786] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8951f8db0, session_id:3221584076, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:203}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.150810] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221584077, support_ssl=false) [2025-01-08 00:51:27.150839] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221584077, sess.client_addr_="127.0.0.1:51682") [2025-01-08 00:51:27.150850] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8951fa030, session_id:3221584077, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:204}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.150914] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A68-0-0] [lt=61] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584070, proxy_sessid=0, client_sessid=3221584070, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.151003] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A69-0-0] [lt=27] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.151772] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A69-0-0] [lt=11] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584071, proxy_sessid=0, client_sessid=3221584071, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.153840] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.153886] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=24] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.153956] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.154002] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=27] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.154022] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.154078] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=35] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.154098] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.154120] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.154134] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.154152] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.154297] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7C-0-0] [lt=27] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.155367] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7C-0-0] [lt=17] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584069, proxy_sessid=0, client_sessid=3221584069, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.155472] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7D-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.156310] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7D-0-0] [lt=14] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584068, proxy_sessid=0, client_sessid=3221584068, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.156417] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7E-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.157064] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7E-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584067, proxy_sessid=0, client_sessid=3221584067, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.157159] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7F-0-0] [lt=22] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.157921] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A7F-0-0] [lt=71] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584066, proxy_sessid=0, client_sessid=3221584066, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.158027] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A80-0-0] [lt=22] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.158876] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A80-0-0] [lt=56] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584065, proxy_sessid=0, client_sessid=3221584065, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.159685] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857AC-0-0] [lt=20] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11008, add_timestamp:1736326287159676}) [2025-01-08 00:51:27.160575] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857AE-0-0] [lt=20] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10008, add_timestamp:1736326287160571}) [2025-01-08 00:51:27.164376] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857AE-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.164843] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857AF-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11017, add_timestamp:1736326287164836}) [2025-01-08 00:51:27.167944] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857B0-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12276, add_timestamp:1736326287167935}) [2025-01-08 00:51:27.171782] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857AF-0-0] [lt=21][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:27.174374] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285980-0-0] [lt=21] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12276, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287174344, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287174344, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287174344, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.174428] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285980-0-0] [lt=53] success to process renew task(task={tenant_id:1, table_id:12276, add_timestamp:1736326287148036}, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287174344, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287174344, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287174344, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.174485] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285981-0-0] [lt=29] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:27.174748] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CDD-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12217, add_timestamp:1736326287174743}) [2025-01-08 00:51:27.174785] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CDD-0-0] [lt=20] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12142, add_timestamp:1736326287174784}) [2025-01-08 00:51:27.174800] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CDD-0-0] [lt=10] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11035, add_timestamp:1736326287174799}) [2025-01-08 00:51:27.174814] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CDD-0-0] [lt=8] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12072, add_timestamp:1736326287174813}) [2025-01-08 00:51:27.174827] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CDD-0-0] [lt=8] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12068, add_timestamp:1736326287174826}) [2025-01-08 00:51:27.174864] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.174969] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.174996] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.175033] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.175068] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.175112] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.175133] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.175235] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=83] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.175310] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.175338] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.175355] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:27.175372] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:27.176061] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CDE-0-0] [lt=17] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11003, add_timestamp:1736326287176057}) [2025-01-08 00:51:27.179087] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe0a030, session_id:3221584069, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326287167341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179127] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221584069, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.179138] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8030, session_id:3221584065, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326287158499, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179338] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=190] kill and revert session(conn.sessid_=3221584065, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.179352] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffc030, session_id:3221584067, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326287158499, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179370] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221584067, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.179377] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8db0, session_id:3221584066, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326287158499, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179392] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] kill and revert session(conn.sessid_=3221584066, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.179398] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffcdb0, session_id:3221584068, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326287158499, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179413] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221584068, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.179420] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8efe0a030, session_id:3221584069, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326287167341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179469] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221584069, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.179482] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221584069, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857B0-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:368, rec_:368, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.179521] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=36] sock ref clean, do destroy(*s={this:0x2ad8efff8030, session_id:3221584065, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326287158499, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179551] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221584065, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.179560] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221584065, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857AC-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:133, rec_:133, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:31, seq_:37}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:38}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.179587] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] sock ref clean, do destroy(*s={this:0x2ad8efffc030, session_id:3221584067, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326287158499, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179629] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221584067, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.179640] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221584067, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857AE-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.179723] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=79] sock ref clean, do destroy(*s={this:0x2ad8efff8db0, session_id:3221584066, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326287158499, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179763] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221584066, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.179772] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221584066, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857AD-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:134, rec_:134, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:7}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.179800] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] sock ref clean, do destroy(*s={this:0x2ad8efffcdb0, session_id:3221584068, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326287158499, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.179830] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221584068, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.179839] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221584068, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857AF-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:549, rec_:549, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:12, seq_:19}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:20}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.179939] INFO [SERVER] run (obmp_disconnect.cpp:83) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=27] free session successfully(sessid=3221584069) [2025-01-08 00:51:27.179961] INFO [SERVER] run (obmp_disconnect.cpp:90) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] mark session id unused(sessid=3221584069) [2025-01-08 00:51:27.180005] INFO [SERVER] run (obmp_disconnect.cpp:83) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221584065) [2025-01-08 00:51:27.180015] INFO [SERVER] run (obmp_disconnect.cpp:90) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221584065) [2025-01-08 00:51:27.180057] INFO [SERVER] run (obmp_disconnect.cpp:83) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221584067) [2025-01-08 00:51:27.180067] INFO [SERVER] run (obmp_disconnect.cpp:90) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221584067) [2025-01-08 00:51:27.180110] INFO [SERVER] run (obmp_disconnect.cpp:83) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221584066) [2025-01-08 00:51:27.180120] INFO [SERVER] run (obmp_disconnect.cpp:90) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221584066) [2025-01-08 00:51:27.180234] INFO [SERVER] run (obmp_disconnect.cpp:83) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221584068) [2025-01-08 00:51:27.180248] INFO [SERVER] run (obmp_disconnect.cpp:90) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=14] mark session id unused(sessid=3221584068) [2025-01-08 00:51:27.185306] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A81-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.186084] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A81-0-0] [lt=23] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584077, proxy_sessid=0, client_sessid=3221584077, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.186251] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A82-0-0] [lt=28] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.186984] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A82-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584076, proxy_sessid=0, client_sessid=3221584076, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.187083] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A83-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.187795] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A83-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584075, proxy_sessid=0, client_sessid=3221584075, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.187894] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A84-0-0] [lt=22] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.188675] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A84-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584074, proxy_sessid=0, client_sessid=3221584074, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.188754] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A85-0-0] [lt=22] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.189564] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A85-0-0] [lt=11] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584073, proxy_sessid=0, client_sessid=3221584073, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.189653] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A86-0-0] [lt=26] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:27.190305] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=20] ====== check clog disk timer task ====== [2025-01-08 00:51:27.190337] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=16] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:27.190387] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A86-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221584072, proxy_sessid=0, client_sessid=3221584072, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:27.190620] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285981-0-0] [lt=16] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11008, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287190591, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287190591, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287190591, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.190660] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285981-0-0] [lt=39] success to process renew task(task={tenant_id:1, table_id:11008, add_timestamp:1736326287159676}, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287190591, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287190591, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287190591, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.190703] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285982-0-0] [lt=26] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=10008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:27.191178] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285982-0-0] [lt=102] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10008, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287191166, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287191166, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287191166, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.191274] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285982-0-0] [lt=95] success to process renew task(task={tenant_id:1, table_id:10008, add_timestamp:1736326287160571}, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287191166, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287191166, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287191166, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.191307] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285983-0-0] [lt=24] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11017, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:27.201769] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285983-0-0] [lt=9] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11017, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287201742, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287201742, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287201742, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.201836] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285983-0-0] [lt=65] success to process renew task(task={tenant_id:1, table_id:11017, add_timestamp:1736326287164836}, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287201742, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287201742, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287201742, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.201891] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285984-0-0] [lt=30] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12276, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:27.202487] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2188) [2025-01-08 00:51:27.205393] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285984-0-0] [lt=24] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12276, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205376, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205376, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205376, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.205430] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285984-0-0] [lt=37] success to process renew task(task={tenant_id:1, table_id:12276, add_timestamp:1736326287167935}, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205376, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205376, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205376, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.205471] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285985-0-0] [lt=25] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12217, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205465, sql_renew_time:1736326287205465, is_mark_fail:false}]) [2025-01-08 00:51:27.205486] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285985-0-0] [lt=15] success to process renew task(task={tenant_id:1, table_id:12217, add_timestamp:1736326287174743}, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205465, sql_renew_time:1736326287205465, is_mark_fail:false}]) [2025-01-08 00:51:27.205506] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285986-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12142, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205502, sql_renew_time:1736326287205502, is_mark_fail:false}]) [2025-01-08 00:51:27.205615] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285986-0-0] [lt=107] success to process renew task(task={tenant_id:1, table_id:12142, add_timestamp:1736326287174784}, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205502, sql_renew_time:1736326287205502, is_mark_fail:false}]) [2025-01-08 00:51:27.205645] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285987-0-0] [lt=14] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11035, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205636, sql_renew_time:1736326287205636, is_mark_fail:false}]) [2025-01-08 00:51:27.205661] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285987-0-0] [lt=15] success to process renew task(task={tenant_id:1, table_id:11035, add_timestamp:1736326287174799}, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205636, sql_renew_time:1736326287205636, is_mark_fail:false}]) [2025-01-08 00:51:27.205681] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285988-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12072, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205676, sql_renew_time:1736326287205676, is_mark_fail:false}]) [2025-01-08 00:51:27.205695] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285988-0-0] [lt=14] success to process renew task(task={tenant_id:1, table_id:12072, add_timestamp:1736326287174813}, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205676, sql_renew_time:1736326287205676, is_mark_fail:false}]) [2025-01-08 00:51:27.205714] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285989-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12068, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205710, sql_renew_time:1736326287205710, is_mark_fail:false}]) [2025-01-08 00:51:27.205728] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285989-0-0] [lt=13] success to process renew task(task={tenant_id:1, table_id:12068, add_timestamp:1736326287174826}, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205710, sql_renew_time:1736326287205710, is_mark_fail:false}]) [2025-01-08 00:51:27.205746] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598A-0-0] [lt=12] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11003, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:27.208418] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=7524, avg_log_submit_to_slide_time=9195) [2025-01-08 00:51:27.209924] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.210732] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=86, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:27.210761] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10367) [2025-01-08 00:51:27.211738] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2181442}], scn_array=[cnt:1, {val:1736326286874645003, v:0}], curr_size=122, accum_size=122, time_cost=10827) [2025-01-08 00:51:27.211798] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2188, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:27.215821] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598A-0-0] [lt=56] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11003, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287215732, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287215732, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287215732, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.215900] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598A-0-0] [lt=78] success to process renew task(task={tenant_id:1, table_id:11003, add_timestamp:1736326287176057}, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287215732, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287215732, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287215732, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.224165] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=63][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe0c030, session_id:3221584070, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326287172081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.224346] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=170] kill and revert session(conn.sessid_=3221584070, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.224362] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sock ref clean, do destroy(*s={this:0x2ad8efe0c030, session_id:3221584070, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326287172081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.224412] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221584070, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.224478] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=65] connection close(sessid=3221584070, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CDE-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:98, rec_:98, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:10}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:11}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.224608] INFO [SERVER] run (obmp_disconnect.cpp:83) [46388][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] free session successfully(sessid=3221584070) [2025-01-08 00:51:27.224628] INFO [SERVER] run (obmp_disconnect.cpp:90) [46388][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=19] mark session id unused(sessid=3221584070) [2025-01-08 00:51:27.229531] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B88-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.230860] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B89-0-0] [lt=159][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:27.231861] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B8B-0-0] [lt=24] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:27.231907] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46491][T1_L0_G0][T1001][YB42C0A89F89-00062B2D1B484B8B-0-0] [lt=16] get_stable_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:27.231933] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46491][T1_L0_G0][T1002][YB42C0A89F89-00062B2D1B484B8B-0-0] [lt=10] get_stable_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:27.234927] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B8A-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.234938] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B8B-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.235078] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B8A-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.236852] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=56] ====== traversal_flush timer task ====== [2025-01-08 00:51:27.236980] INFO [MDS] flush (mds_table_mgr.cpp:195) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854F2-0-0] [lt=38] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326286874645003, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326283797570004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326300087570000, latest_log_ts:{val:1736326283797570001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.237109] INFO [MDS] flush (mds_table_mgr.cpp:172) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854F2-0-0] [lt=96] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326281848053002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326283797570004, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.237253] INFO [MDS] flush (mds_table_mgr.cpp:172) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854F2-0-0] [lt=57] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326281655245003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326283797570004, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.237346] INFO [MDS] flush (mds_table_mgr.cpp:172) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854F2-0-0] [lt=63] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326281783436004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326283797570004, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.237403] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854F2-0-0] [lt=53] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:27.238201] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AAF-0-0] [lt=6] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11021, add_timestamp:1736326287238185}) [2025-01-08 00:51:27.245362] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_stat.cpp:301) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AAF-0-0] [lt=88][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:27.245843] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB0-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.248877] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68CD-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=5092) [2025-01-08 00:51:27.249003] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598B-0-0] [lt=344] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11021, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:27.249850] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598B-0-0] [lt=38] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11021, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287249828, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287249828, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287249828, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.249911] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598B-0-0] [lt=59] success to process renew task(task={tenant_id:1, table_id:11021, add_timestamp:1736326287238185}, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287249828, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287249828, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287249828, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:27.258156] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A7A-0-0] [lt=38] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326287258092}) [2025-01-08 00:51:27.267052] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598C-0-0] [lt=28] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287267027, sql_renew_time:1736326287267027, is_mark_fail:false}]) [2025-01-08 00:51:27.267116] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598C-0-0] [lt=62] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326287258092}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287267027, sql_renew_time:1736326287267027, is_mark_fail:false}]) [2025-01-08 00:51:27.273524] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15142363883, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:27.273590] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:27.273611] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=20] try_wait return eagain(this=0x5572d527f900, thread=0x2ad8d42059c0) [2025-01-08 00:51:27.273714] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=101] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15142363883, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:27.273739] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=24] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7873132649, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:27.273761] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:27.273778] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try_wait return eagain(this=0x2ad8c0eb0e70, thread=0x2ad9375f5e90) [2025-01-08 00:51:27.273793] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7873132649, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:27.273994] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8951f8030, session_id:3221584075, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326287223113, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.274065] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=56] kill and revert session(conn.sessid_=3221584075, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.274086] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8951f6030, session_id:3221584073, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326287223113, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.274433] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=335] kill and revert session(conn.sessid_=3221584073, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.274454] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dfc030, session_id:3221584072, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326287223113, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.274488] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221584072, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.274500] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8951f6db0, session_id:3221584074, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326287223113, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.274531] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221584074, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.274546] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] sock ref clean, do destroy(*s={this:0x2ad8951f8030, session_id:3221584075, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326287223113, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.274622] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] unlock session of tenant(conn.sessid_=3221584075, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.274643] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] connection close(sessid=3221584075, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B8B-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:562, rec_:562, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.274708] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=60] sock ref clean, do destroy(*s={this:0x2ad8951f6030, session_id:3221584073, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326287223113, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.274774] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] unlock session of tenant(conn.sessid_=3221584073, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.274795] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] connection close(sessid=3221584073, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B89-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:241, rec_:241, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.274852] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=52] sock ref clean, do destroy(*s={this:0x2ad8a0dfc030, session_id:3221584072, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326287223113, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.274907] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] unlock session of tenant(conn.sessid_=3221584072, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.274928] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] connection close(sessid=3221584072, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B88-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:436, rec_:436, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.274969] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=38] sock ref clean, do destroy(*s={this:0x2ad8951f6db0, session_id:3221584074, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326287223113, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.275014] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] unlock session of tenant(conn.sessid_=3221584074, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.275027] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221584074, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B8A-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.275260] INFO [SERVER] run (obmp_disconnect.cpp:83) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=92] free session successfully(sessid=3221584075) [2025-01-08 00:51:27.275371] INFO [SERVER] run (obmp_disconnect.cpp:90) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=106] mark session id unused(sessid=3221584075) [2025-01-08 00:51:27.275560] INFO [SERVER] run (obmp_disconnect.cpp:83) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=22] free session successfully(sessid=3221584073) [2025-01-08 00:51:27.275584] INFO [SERVER] run (obmp_disconnect.cpp:90) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=22] mark session id unused(sessid=3221584073) [2025-01-08 00:51:27.275658] INFO [SERVER] run (obmp_disconnect.cpp:83) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=11] free session successfully(sessid=3221584072) [2025-01-08 00:51:27.275677] INFO [SERVER] run (obmp_disconnect.cpp:90) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=18] mark session id unused(sessid=3221584072) [2025-01-08 00:51:27.275745] INFO [SERVER] run (obmp_disconnect.cpp:83) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] free session successfully(sessid=3221584074) [2025-01-08 00:51:27.275762] INFO [SERVER] run (obmp_disconnect.cpp:90) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=16] mark session id unused(sessid=3221584074) [2025-01-08 00:51:27.277883] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:27.278111] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8951fa030, session_id:3221584077, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326287237955, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.278150] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221584077, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.278159] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8951f8db0, session_id:3221584076, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326287237955, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.278212] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=45] kill and revert session(conn.sessid_=3221584076, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.278338] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=124] sock ref clean, do destroy(*s={this:0x2ad8951fa030, session_id:3221584077, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326287237955, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.278692] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=316] unlock session of tenant(conn.sessid_=3221584077, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.278714] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] connection close(sessid=3221584077, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AB0-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:633, rec_:633, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.278767] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=49] sock ref clean, do destroy(*s={this:0x2ad8951f8db0, session_id:3221584076, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326287237955, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.278930] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] unlock session of tenant(conn.sessid_=3221584076, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.278951] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] connection close(sessid=3221584076, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AAF-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:688, rec_:688, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:219, seq_:225}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:226}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.279572] INFO [SERVER] run (obmp_disconnect.cpp:83) [46384][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=133] free session successfully(sessid=3221584077) [2025-01-08 00:51:27.279604] INFO [SERVER] run (obmp_disconnect.cpp:90) [46384][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=31] mark session id unused(sessid=3221584077) [2025-01-08 00:51:27.279660] INFO [SERVER] run (obmp_disconnect.cpp:83) [46384][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] free session successfully(sessid=3221584076) [2025-01-08 00:51:27.279671] INFO [SERVER] run (obmp_disconnect.cpp:90) [46384][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=11] mark session id unused(sessid=3221584076) [2025-01-08 00:51:27.282581] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=86] ====== check clog disk timer task ====== [2025-01-08 00:51:27.282648] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=53] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:27.287791] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=206] start to print blacklist info [2025-01-08 00:51:27.288472] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=48] ls blacklist refresh finish(cost_time=32646) [2025-01-08 00:51:27.295555] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe0adb0, session_id:3221584071, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326287172081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.295613] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=49] kill and revert session(conn.sessid_=3221584071, proxy_sessid=0, ret=0) [2025-01-08 00:51:27.295626] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] sock ref clean, do destroy(*s={this:0x2ad8efe0adb0, session_id:3221584071, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326287172081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:27.295666] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221584071, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:27.295678] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221584071, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CDD-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:119, rec_:119, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:2, seq_:4}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:5}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:27.295893] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=23] free session successfully(sessid=3221584071) [2025-01-08 00:51:27.295972] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=77] mark session id unused(sessid=3221584071) [2025-01-08 00:51:27.304847] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58538][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:27.305035] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:27.305616] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:27.306502] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58539][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:27.306559] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC392D-0-0] [lt=23] init point thread id with(&point=0x5572d50e5400, idx_=1195, point=[thread id=58539, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58539) [2025-01-08 00:51:27.306606] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:27.306751] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11] ts response task statistics(alloc_count=60266, free_count=30132) [2025-01-08 00:51:27.306835] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:27.309526] INFO [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:174) [46345][T1_ArcSeq][T1][YB42C0A89F89-00062B2D1668458E-0-0] [lt=21] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-01-08 00:51:27.311023] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.311835] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8A1-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1736326287308713000, start_id=1736326287308713000, end_id=1736326287308713001) [2025-01-08 00:51:27.314328] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE8A0-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326287313717000, start_id=1736326287313717000, end_id=1736326287313717001) [2025-01-08 00:51:27.314545] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2219287, this={dio_aligned_buf:{buf_write_offset:3473, buf_padding_size:623, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:26553, truncate_used_ts:5468}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2215936, buf_len=122, total_write_size=10586, total_write_size_after_dio=360448, ob_pwrite_used_ts=1016243, count=86, each_pwrite_cost=11816) [2025-01-08 00:51:27.319462] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=93] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.320840] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:27.321050] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%", tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-01-08 00:51:27.321262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=65] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.329872] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D43-0-0] [lt=37] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:27.341534] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:27.343124] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=68] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:27.346318] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:27.346575] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:27.348091] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:27.355636] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=49] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:27.370119] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=54] PNIO [ratelimit] time: 1736326287370116, bytes: 876772975, bw: 0.301627 MB/s, add_ts: 1000638, add_bytes: 316481 [2025-01-08 00:51:27.378414] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=14] current monitor number(seq_=-1) [2025-01-08 00:51:27.383753] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBD-0-0] [lt=534] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:27.383792] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBD-0-0] [lt=35] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:27.408937] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=73] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5092) [2025-01-08 00:51:27.409031] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=87] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:27.409850] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=74] ====== tenant freeze timer task ====== [2025-01-08 00:51:27.411440] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=371][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.413683] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB604-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:27.413750] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB604-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326286813632003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326286813632003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:27.413980] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5555) [2025-01-08 00:51:27.415695] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68CE-0-0] [lt=41] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:27.415796] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68CE-0-0] [lt=100] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8291, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:27.415813] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68CE-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=93, avg_log_submit_to_flush_time=4489, avg_log_submit_to_slide_time=7556) [2025-01-08 00:51:27.416072] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=55] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5555, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:27.416717] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=32] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:27.418265] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:27.418292] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70914440, tenant_id=1001) [2025-01-08 00:51:27.418315] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:27.423929] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] ====== traversal_flush timer task ====== [2025-01-08 00:51:27.424079] INFO [MDS] flush (mds_table_mgr.cpp:195) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F1C-0-0] [lt=37] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326287410958000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299284252000, latest_log_ts:{val:1736326282955407001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.424264] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F1C-0-0] [lt=177] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:27.427964] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:149) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB604-0-0] [lt=38] transaction rpc statistics(total_rt=7144, total_process=27, avg_rt=255) [2025-01-08 00:51:27.430932] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=90] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:27.448606] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8698, offset=2469304}, end_cursor:ObLogCursor{file_id=1, log_id=8699, offset=2469609}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894adc050, len:584, log_data_len:305, seq:8698, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:27.449290] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5B-0-0] [lt=79] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:27.449364] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5B-0-0] [lt=50] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326287305763000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326287305763000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:27.454782] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E5B-0-0] [lt=9] get number(ret=0, service_type_=0, range=1, base_id=1736326287449272000, start_id=1736326287449272000, end_id=1736326287449272001) [2025-01-08 00:51:27.456989] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=120] gc stale ls task succ [2025-01-08 00:51:27.457941] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=67] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:27.458075] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=44] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:27.458101] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:27.458112] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:27.461067] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=132, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}]) [2025-01-08 00:51:27.461547] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=477] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:27.461660] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.461702] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.462869] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58540][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:27.463028] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:51:27.465007] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=61] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8699, offset=2469609}, end_cursor:ObLogCursor{file_id=1, log_id=8700, offset=2469906}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ae0050, len:4375, log_data_len:297, seq:8699, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:27.466779] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5B-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:27.466899] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5B-0-0] [lt=48] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326287305763000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326287305763000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:27.475551] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=46] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8700, offset=2469906}, end_cursor:ObLogCursor{file_id=1, log_id=8701, offset=2470203}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ae4050, len:4078, log_data_len:297, seq:8700, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:27.477419] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=141] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8701, offset=2470203}, end_cursor:ObLogCursor{file_id=1, log_id=8702, offset=2470500}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ae8050, len:3781, log_data_len:297, seq:8701, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:27.478092] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18007, miss_cnt:0, cache_read_size:2202792, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:27.483513] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:27.485406] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58530][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=45] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-01-08 00:51:27.505307] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:27.506793] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=89] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:27.507160] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3931-0-0] [lt=23] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=15437, avg_log_submit_to_slide_time=64811) [2025-01-08 00:51:27.507325] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:27.507683] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4691-0-0] [lt=40][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18201, log_task={header:{begin_lsn:{lsn:2215313}, end_lsn:{lsn:2215435}, log_id:18201, min_scn:{val:1736326287318696000, v:0}, max_scn:{val:1736326287318696000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2215191}, prev_proposal_id:1, committed_end_lsn:{lsn:2215313}, data_checksum:1715074492, accum_checksum:3632591099, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326287407562, freeze_ts:1736326287407562, submit_ts:1736326287407566, flushed_ts:1736326287413932, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:6366}, fs_cb_begin_ts=1736326287507675, log_life_time=100113) [2025-01-08 00:51:27.509713] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9326-0-0] [lt=55] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:27.509806] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9326-0-0] [lt=91] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54659, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:27.511507] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=117][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.516561] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AD25-0-0] [lt=25] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.840151977539062500e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=392, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.431660989222258422e+02) [2025-01-08 00:51:27.519510] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.541707] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.542015] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=73, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:27.542068] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=4424, avg_group_log_size=442) [2025-01-08 00:51:27.543901] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.598317] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:27.598410] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=90] try gc retain ctx [2025-01-08 00:51:27.598419] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:27.598504] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326287305763001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326277123056004, v:0}, keep_alive_scn_:{val:1736326277123056004, v:0}, update_ts_:1736326287597825}) [2025-01-08 00:51:27.599358] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326284800559003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326300087570000, latest_log_ts:{val:1736326283797570001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:27.599646] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=137] finish update min start scn(max_decided_scn={val:1736326287305763001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326277123056004, v:0}, keep_alive_scn_:{val:1736326277123056004, v:0}, update_ts_:1736326287597825}) [2025-01-08 00:51:27.600751] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=88] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326281848053002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326284800559003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:27.601034] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=119] finish update min start scn(max_decided_scn={val:1736326287318696001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326280694597000, v:0}, keep_alive_scn_:{val:1736326285326952004, v:0}, update_ts_:1736326287597825}) [2025-01-08 00:51:27.602296] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326281655245003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326284800559003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:27.602366] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=62] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:27.602385] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] get_min_unreplayed_log_info(lsn={lsn:2203036}, scn={val:1736326287308713002, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326287509693, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2203036}, next_to_submit_scn_:{val:1736326287308713002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2202914}, end_lsn:{lsn:2203036}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2203036}, readable_log_tail:{lsn:2203036}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64901732, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326287308713001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326287308713001, v:0}, data_checksum:2124447694, flag:52127}}, init_mode_version:0, accumulate_checksum:2949852216, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326287308713001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18009, miss_cnt:0, cache_read_size:2203036, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:27.602458] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=72] finish update min start scn(max_decided_scn={val:1736326287308713001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326279275750001, v:0}, keep_alive_scn_:{val:1736326284308664001, v:0}, update_ts_:1736326287597825}) [2025-01-08 00:51:27.603523] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326281783436004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326284800559003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:27.613376] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:27.613408] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.617149] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=85][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18693098}], scn_array=[cnt:1, {val:1736326287305763002, v:0}], curr_size=113, accum_size=479, time_cost=13008) [2025-01-08 00:51:27.698470] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:27.698537] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, bw:40832, [delay/us]:prepare:17, schedule:904, submit:577, rt:2040, total:3539) [2025-01-08 00:51:27.698554] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:27.698612] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40843, [delay/us]: prepare:17, schedule:904, submit:577, device:2040, total:3539) [2025-01-08 00:51:27.698625] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:27.698708] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:14, bw:65354, [delay/us]:prepare:11, schedule:616, submit:611, rt:3023, total:4263) [2025-01-08 00:51:27.698722] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:27.698760] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:14, avg_bw:65351, [delay/us]: prepare:11, schedule:616, submit:611, device:3023, total:4263) [2025-01-08 00:51:27.698771] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:27.698795] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20425, [delay/us]:prepare:8, schedule:132, submit:1, rt:1458, total:1600) [2025-01-08 00:51:27.698817] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:38, bw:155232, [delay/us]:prepare:9, schedule:747, submit:376, rt:3214, total:4348) [2025-01-08 00:51:27.698867] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:27.698901] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20426, [delay/us]: prepare:8, schedule:132, submit:1, device:1458, total:1600) [2025-01-08 00:51:27.698917] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:38, avg_bw:155244, [delay/us]: prepare:9, schedule:747, submit:376, device:3214, total:4348) [2025-01-08 00:51:27.698928] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:27.698948] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4085, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:428, total:439) [2025-01-08 00:51:27.698972] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:27.698999] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:428, total:439) [2025-01-08 00:51:27.699017] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:27.699085] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:27.699094] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:27.699102] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:27.699110] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=240kB/s, net_out=227kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:27.705694] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=125] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:27.708542] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:27.710698] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84987-0-0] [lt=17] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:27.710653] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=93] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:27.710776] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=121] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:27.710782] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84987-0-0] [lt=78] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:27.711104] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708515F-0-0] [lt=16] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:27.713493] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.727792] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AE-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:27.727851] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AE-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326287318696003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326287318696003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:27.780409] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=27] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:27.794606] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] ====== check clog disk timer task ====== [2025-01-08 00:51:27.794642] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=29] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:27.800670] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:27.800877] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=74][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%", tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:27.801328] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=31] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:27.802559] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1}, {id:1003}]) [2025-01-08 00:51:27.802708] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:27.802745] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:51:27.802839] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:51:27.806266] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:27.806291] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:27.807524] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17] get number(ret=0, service_type_=0, range=1, base_id=1736326287807361000, start_id=1736326287807361000, end_id=1736326287807361001) [2025-01-08 00:51:27.807590] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=39] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326287807514], gts:1736326287807361000, latest_srr:[mts=1736326287807514]}) [2025-01-08 00:51:27.807623] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326287807610], gts:1736326287807361000, latest_srr:[mts=1736326287807610]}) [2025-01-08 00:51:27.807912] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326287807898], gts:1736326287807361000, latest_srr:[mts=1736326287807898]}) [2025-01-08 00:51:27.815338] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=100][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.822843] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=26] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:27.824483] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F538-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326287819655000, start_id=1736326287819655000, end_id=1736326287819655001) [2025-01-08 00:51:27.827418] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857B1-0-0] [lt=5][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193792741251, cache_obj->added_lc()=false, cache_obj->get_object_id()=13639, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:27.828945] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185432-0-0] [lt=14][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.832903] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857B2-0-0] [lt=10][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:27.834720] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870DC-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.834769] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:27.834781] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:27.837349] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:27.837385] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=24] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=93) [2025-01-08 00:51:27.837494] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3D4-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.840336] WDIAG get_refactored (ob_hashtable.h:1120) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857B4-0-0] [lt=8][errcode=-4201] hashtable not init [2025-01-08 00:51:27.841856] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:27.842088] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D06E-0-0] [lt=5][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.843133] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=52] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:27.843349] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=69] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:27.846756] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:27.846887] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:27.849968] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:27.855976] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=139] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:27.870880] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=41] ====== traversal_flush timer task ====== [2025-01-08 00:51:27.871096] INFO [MDS] flush (mds_table_mgr.cpp:195) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856EA-0-0] [lt=93] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326287815736000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299254996000, latest_log_ts:{val:1736326282603912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.871267] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856EA-0-0] [lt=163] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:27.872491] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=40] PNIO [ratelimit] time: 1736326287872487, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000685, add_bytes: 0 [2025-01-08 00:51:27.877776] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=17] ====== checkpoint timer task ====== [2025-01-08 00:51:27.878841] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=43] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:27.878910] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=68] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:27.878980] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326284800559003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326300087570000, latest_log_ts:{val:1736326283797570001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.879077] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=93] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:27.879100] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:27.879115] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326283797570001, v:0}) [2025-01-08 00:51:27.879122] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:27.879136] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:27.879153] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:27.879164] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:27.879621] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=444][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:27.879649] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:27.880654] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:27.880698] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=43] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:27.880740] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326281848053002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326284800559003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.880802] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=60] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:27.880815] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:27.880829] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:27.880844] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:27.880860] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2182287}, scn={val:1736326287305763004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:27.881265] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=62] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:284K/66:458271 disk:284K/66:458271 t1m2:36K/8:102979 t1m2g0:36K/8:102979 t1002m2:172K/42:598980 t1002m2g0:172K/42:598980 t1001m2:76K/16:263860 t1001m2g0:76K/16:263860 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:27.883265] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=105] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8702, offset=2470500}, end_cursor:ObLogCursor{file_id=1, log_id=8703, offset=2470797}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894aec050, len:3484, log_data_len:297, seq:8702, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:27.883336] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326287305763004, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:27.883364] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:27.883385] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:27.884144] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:27.884177] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=32] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:27.884291] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326281655245003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326284800559003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.884357] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=64] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:27.884369] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:27.884383] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:27.884397] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:27.884411] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2215923}, scn={val:1736326287318696004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:27.884789] INFO [PALF] pread (log_reader.cpp:130) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1696) [2025-01-08 00:51:27.887807] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8703, offset=2470797}, end_cursor:ObLogCursor{file_id=1, log_id=8704, offset=2471094}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894af0050, len:3187, log_data_len:297, seq:8703, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:27.887994] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326287318696004, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:27.888025] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:27.888045] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:27.888834] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:27.888866] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=33] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:27.888907] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326281783436004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326284800559003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:27.888968] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=59] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:27.888980] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:27.888993] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:27.889007] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:27.889021] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2203280}, scn={val:1736326287308713003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:27.889384] INFO [PALF] pread (log_reader.cpp:130) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=345) [2025-01-08 00:51:27.890319] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8704, offset=2471094}, end_cursor:ObLogCursor{file_id=1, log_id=8705, offset=2471391}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894af4050, len:2890, log_data_len:297, seq:8704, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:27.890383] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326287308713003, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:27.890411] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:27.890431] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:27.890440] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:27.896731] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.896760] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.896787] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.897560] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:27.908942] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=110] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:27.908991] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=103] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1321152, replace_num=15728) [2025-01-08 00:51:27.909018] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:27.915514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:27.918558] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:27.923682] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=38] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:27.933856] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA6-0-0] [lt=80][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:27.936119] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA6-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:28.002132] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=74] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=73, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:28.002176] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:28.002197] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:28.002211] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:28.002384] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=124] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=47, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:28.002416] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:28.002430] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:28.002439] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:28.006696] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:28.007105] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:28.009906] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4229, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:28.009964] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1140, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:28.010078] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:28.010113] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=123, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:28.010138] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=4, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:28.013667] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=54] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:28.014403] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9335-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:28.014824] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=10064, avg_group_log_size=592) [2025-01-08 00:51:28.014854] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:28.015750] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.018043] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=49] [PALF STAT WRITE LOG COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=3816, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:28.018085] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10064) [2025-01-08 00:51:28.018108] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=349, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:28.018223] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10064) [2025-01-08 00:51:28.018272] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=25, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:28.022843] INFO [LIB] stat (utility.h:1210) [58533][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A1C-0-0] [lt=46] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=11, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:28.022885] INFO [LIB] stat (utility.h:1210) [58533][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A1C-0-0] [lt=41] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=7806, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:28.022898] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58533][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A1C-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=316, avg_log_gen_to_submit_time=416, avg_log_submit_to_flush_time=4221, avg_log_submit_to_slide_time=7390) [2025-01-08 00:51:28.023048] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10064, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:28.024381] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326287305763002, v:0}, new_max_version={val:1736326287305763002, v:0}, last_min_version={val:1736326287305763001, v:0}, last_max_version={val:1736326287305763001, v:0}, total_time_us=12580, delta_us=718616) [2025-01-08 00:51:28.024439] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=53] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326287305763002, v:0}, min_version={val:1736326287305763002, v:0}, max_version={val:1736326287305763002, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:28.076714] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=33] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5261655658, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:28.076927] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=118] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:28.076951] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=22] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5261655658, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:28.105777] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9336-0-0] [lt=35] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:28.105812] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9336-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9167, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:28.107055] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC393D-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:28.107333] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC393D-0-0] [lt=256] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:28.109159] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:28.109465] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:28.110418] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:28.110898] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A469D-0-0] [lt=103] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:28.110945] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A469D-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:28.110962] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A469D-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:28.110996] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A469D-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:28.111007] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A469D-0-0] [lt=11] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=60508, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:28.111015] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A469D-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=20, avg_log_submit_to_flush_time=4151, avg_log_submit_to_slide_time=60487) [2025-01-08 00:51:28.111026] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A469D-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18207, log_task={header:{begin_lsn:{lsn:2216045}, end_lsn:{lsn:2216167}, log_id:18207, min_scn:{val:1736326287819655001, v:0}, max_scn:{val:1736326287819655001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2215923}, prev_proposal_id:1, committed_end_lsn:{lsn:2216045}, data_checksum:1715074492, accum_checksum:551247531, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326288009899, freeze_ts:1736326288009899, submit_ts:1736326288009905, flushed_ts:1736326288017058, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:7153}, fs_cb_begin_ts=1736326288110975, log_life_time=101076) [2025-01-08 00:51:28.111050] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:28.111147] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=88] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18115, miss_cnt:0, cache_read_size:2216167, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:28.112531] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3758, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:28.112558] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:28.112573] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=262, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:28.112660] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=44, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:28.115962] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.120898] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8D-0-0] [lt=63] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326288231583, keepalive_ts:-1}) [2025-01-08 00:51:28.121004] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8D-0-0] [lt=52] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326288231583, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:28.131818] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C672-0-0] [lt=20][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193793045651, cache_obj->added_lc()=false, cache_obj->get_object_id()=13640, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:28.145496] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=89] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2663, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:28.145606] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=101] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=668, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:28.145668] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:28.170859] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=17][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:28.171003] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=24][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:28.171129] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=12][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:28.171651] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326278109964, end_time=1736326288171648, dag_status={}) [2025-01-08 00:51:28.171684] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=30] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:28.171697] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:28.171706] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:28.171713] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:28.171720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:28.171727] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:28.171734] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:28.171740] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:28.171758] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:28.171767] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171778] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171786] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171794] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171802] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171810] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171818] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171826] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171834] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171842] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171850] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171858] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171866] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171874] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171882] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171890] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171897] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171905] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171913] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171921] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171929] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171937] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171945] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171952] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171960] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171968] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171976] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171984] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171991] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.171999] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172007] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172015] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172023] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172030] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172038] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172046] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172054] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172061] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172069] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172085] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172093] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172101] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172109] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172117] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172124] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172132] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172140] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172147] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172155] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172163] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172171] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172179] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172186] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172194] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172268] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=73] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172277] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172285] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172293] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172300] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172308] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172316] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172331] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172339] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172347] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172355] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172363] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172370] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172378] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172386] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172394] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172401] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:28.172450] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=47] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:28.172460] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:28.172467] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:28.172472] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:28.172478] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:28.172483] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:28.172488] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:28.172493] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:28.172498] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:28.172503] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:28.172509] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:28.203573] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2747, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:28.203642] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=66] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1333) [2025-01-08 00:51:28.203670] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=394, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:28.203708] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:28.203994] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=154] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3463, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:28.204025] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:28.204066] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=646, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:28.204136] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=67] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=28, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:28.205955] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9339-0-0] [lt=117] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=42, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:28.205978] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9339-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6467, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:28.206088] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:28.214621] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=88, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:28.214669] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9640) [2025-01-08 00:51:28.216257] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:28.216291] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.234002] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852A9-0-0] [lt=16] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:28.277227] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15146579243, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:28.277364] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=42] recycle one thread(this=0x5572d527f900, total=5, remain=4) [2025-01-08 00:51:28.277382] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15146579243, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:28.277394] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7877159533, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:28.277422] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] recycle one thread(this=0x2ad8c0eb0e70, total=5, remain=4) [2025-01-08 00:51:28.277430] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7877159533, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:28.278101] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:28.306731] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A933D-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=16, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=3994, avg_log_submit_to_slide_time=7967) [2025-01-08 00:51:28.306972] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:28.307208] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:28.307742] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58465][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=24] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:28.308248] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:28.309965] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:28.310618] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=93] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:28.316395] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.318996] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F53D-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326288318549000, start_id=1736326288318549000, end_id=1736326288318549001) [2025-01-08 00:51:28.319222] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.319377] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.332062] INFO [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:130) [46348][T1_ArcTimer][T1][YB42C0A89F89-00062B2D1698458E-0-0] [lt=62] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-01-08 00:51:28.332174] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:28.332289] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=107] temporary file module use no blocks [2025-01-08 00:51:28.332300] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:28.332430] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=127] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:28.332452] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193792245735, flush_io_finished_ret_:0, flush_io_finished_round_:1951, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:28.342090] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:28.343542] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=47] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:28.346963] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:28.347042] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=48] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:28.350207] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:28.356324] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:28.364636] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=69] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:51:28.364746] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:51:28.366154] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=629149696, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:51:28.366341] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=183] === Run print tenant memory usage task === [2025-01-08 00:51:28.366434] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=16] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,350,656 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 629,149,696 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,288,365,820 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 160,381,740 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,042,788,352 max_mem_memstore_can_get_now= 801,908,736 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,288,365,820 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 71,124,160 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 647,712,768 max_mem_memstore_can_get_now= 355,620,848 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,288,365,820 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 240,672,760 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,114,390,528 max_mem_memstore_can_get_now= 1,203,363,818 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:51:28.366664] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=36] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,657,314,304 total_hold= 3,856,662,528 used= 3,467,284,480 freelists_hold= 190,029,824 total_maps= 1,635 total_unmaps= 243 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,350,656 virtual_memory_used= 4,828,954,624 [CHUNK_MGR] 2 MB_CACHE: hold= 90,177,536 count= 43 pushes= 1,826,474 pops= 1,826,431 maps= 1,504 unmaps= 238 [CHUNK_MGR] 4 MB_CACHE: hold= 7,745,536 count= 3 pushes= 6,154 pops= 6,151 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,114 pops= 3,110 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:51:28.366709] INFO print (ob_malloc_time_monitor.cpp:37) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=29] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 6998, delta_count= 2, avg_cost_time= 3499 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 34141, delta_count= 2, avg_cost_time= 17070 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 111894, delta_count= 1, avg_cost_time= 111894 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-01-08 00:51:28.377792] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=43] PNIO [ratelimit] time: 1736326288377789, bytes: 877024693, bw: 0.238229 MB/s, add_ts: 1007673, add_bytes: 251718 [2025-01-08 00:51:28.378618] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-01-08 00:51:28.385284] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBE-0-0] [lt=14] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:28.385316] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBE-0-0] [lt=29] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:28.416579] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.421631] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=96] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3416) [2025-01-08 00:51:28.421665] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:28.423564] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3417) [2025-01-08 00:51:28.423692] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3416) [2025-01-08 00:51:28.424605] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68D7-0-0] [lt=130] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:28.424626] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68D7-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=6065, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:28.424637] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA68D7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=122, avg_log_submit_to_flush_time=2401, avg_log_submit_to_slide_time=5942) [2025-01-08 00:51:28.424747] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=77] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3417, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:28.428501] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58540][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1C2BB60C-0-0] [lt=52] init point thread id with(&point=0x5572d50e5440, idx_=1196, point=[thread id=58540, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58540) [2025-01-08 00:51:28.438302] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB605-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:28.438410] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB605-0-0] [lt=91] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326288310797000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326288310797000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:28.453650] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=22] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:28.457150] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:51:28.458685] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.458747] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.470628] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:28.472674] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58533][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=13] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:28.477587] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x5572d527f900, last_check_time:1736326288477582, last_idle_time:15147244958, last_thread_count:4}, *pool={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15147244958, tenant_id:500}, qsize=-4, current_idle_time=15147244958) [2025-01-08 00:51:28.477626] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=39] [thread pool stat](pool_stat={pool:0x2ad8885abcf0, last_check_time:1736326288477625, last_idle_time:5262451195, last_thread_count:2}, *pool={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5262451195, tenant_id:500}, qsize=-2, current_idle_time=5262451195) [2025-01-08 00:51:28.477639] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] [thread pool stat](pool_stat={pool:0x2ad8885acaf0, last_check_time:1736326288477639, last_idle_time:3784609100, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885acaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3784609100, tenant_id:500}, qsize=-1, current_idle_time=3784609100) [2025-01-08 00:51:28.477650] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8881fc3b0, last_check_time:1736326288477650, last_idle_time:15075602547, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15075602547, tenant_id:1}, qsize=-4, current_idle_time=15075602547) [2025-01-08 00:51:28.477660] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8885ad1f0, last_check_time:1736326288477660, last_idle_time:3696733823, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885ad1f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3696733823, tenant_id:500}, qsize=-1, current_idle_time=3696733823) [2025-01-08 00:51:28.477670] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8c0eb0e70, last_check_time:1736326288477670, last_idle_time:7877954465, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7877954465, tenant_id:1002}, qsize=-4, current_idle_time=7877954465) [2025-01-08 00:51:28.477680] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8f90b0e70, last_check_time:1736326288477679, last_idle_time:7841221801, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7841221801, tenant_id:1001}, qsize=-4, current_idle_time=7841221801) [2025-01-08 00:51:28.478074] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=40] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:28.482047] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8705, offset=2471391}, end_cursor:ObLogCursor{file_id=1, log_id=8706, offset=2471696}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894af8050, len:2593, log_data_len:305, seq:8705, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:28.482742] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5C-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:28.482805] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5C-0-0] [lt=39] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326288308139000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326288308139000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:28.491096] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:28.491404] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58519][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=37] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:51:28.492140] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=49] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8706, offset=2471696}, end_cursor:ObLogCursor{file_id=1, log_id=8707, offset=2471993}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894afc050, len:2288, log_data_len:297, seq:8706, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:28.492866] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58542][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:28.492915] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:51:28.493129] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5C-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:28.493170] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5C-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326288308139000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326288308139000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:28.500737] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=43] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8707, offset=2471993}, end_cursor:ObLogCursor{file_id=1, log_id=8708, offset=2472290}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b00050, len:1991, log_data_len:297, seq:8707, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:28.502084] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=62] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8708, offset=2472290}, end_cursor:ObLogCursor{file_id=1, log_id=8709, offset=2472587}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b04050, len:1694, log_data_len:297, seq:8708, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:28.502303] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18018, miss_cnt:0, cache_read_size:2204134, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:28.510608] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=117] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:28.511235] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:28.511590] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9344-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:28.511617] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9344-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20504, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:28.511650] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:28.511670] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9344-0-0] [lt=51] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=36, avg_log_submit_to_flush_time=2768, avg_log_submit_to_slide_time=18839) [2025-01-08 00:51:28.516668] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.520954] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.541290] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.543921] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.544840] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=96, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:28.544874] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=3417, avg_group_log_size=379) [2025-01-08 00:51:28.613312] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58543][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:28.615821] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ADB8-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.840151977539062500e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=392, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.431660989222258422e+02) [2025-01-08 00:51:28.620677] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.627969] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:28.642008] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=97] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:28.702939] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:28.703011] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:9, bw:36699, [delay/us]:prepare:7, schedule:186, submit:178, rt:1574, total:1946) [2025-01-08 00:51:28.703084] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=66] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:28.703256] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=155] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:9, avg_bw:36693, [delay/us]: prepare:7, schedule:186, submit:178, device:1574, total:1946) [2025-01-08 00:51:28.703271] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:28.703312] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:81541, [delay/us]:prepare:13, schedule:306, submit:97, rt:2198, total:2616) [2025-01-08 00:51:28.703323] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:28.703361] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:81544, [delay/us]: prepare:13, schedule:306, submit:97, device:2198, total:2616) [2025-01-08 00:51:28.703372] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:28.703393] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:7, bw:28540, [delay/us]:prepare:5, schedule:69, submit:1, rt:1117, total:1193) [2025-01-08 00:51:28.703415] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:40, bw:179399, [delay/us]:prepare:8, schedule:113, submit:146, rt:1554, total:1822) [2025-01-08 00:51:28.703424] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12231, [delay/us]:prepare:2, schedule:92, submit:80, rt:259, total:434) [2025-01-08 00:51:28.703432] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:28.703459] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:7, avg_bw:28541, [delay/us]: prepare:5, schedule:69, submit:1, device:1117, total:1193) [2025-01-08 00:51:28.703475] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12232, [delay/us]: prepare:2, schedule:92, submit:80, device:259, total:434) [2025-01-08 00:51:28.703533] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=56] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:40, avg_bw:179396, [delay/us]: prepare:8, schedule:113, submit:146, device:1554, total:1822) [2025-01-08 00:51:28.703544] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:28.703565] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4077, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1004, total:1017) [2025-01-08 00:51:28.703590] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:28.703620] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4077, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1004, total:1017) [2025-01-08 00:51:28.703638] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:28.703653] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:28.703661] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:28.703669] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:28.703677] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=264kB/s, net_out=218kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:28.710055] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9349-0-0] [lt=39] init point thread id with(&point=0x5572d50e5500, idx_=1199, point=[thread id=58543, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58543) [2025-01-08 00:51:28.710840] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:28.711846] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=102] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:28.715606] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=84] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1736326288308139002, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3, epoch_tstamp:1736326288715426}, version_delta=407464) [2025-01-08 00:51:28.721333] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.728565] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=323] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43341154}, readable_log_tail:{lsn:43341154}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23763614, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:28.737878] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=35] ====== tenant freeze timer task ====== [2025-01-08 00:51:28.741340] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=25] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:28.749221] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB59-0-0] [lt=29] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:28.749259] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB59-0-0] [lt=71] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:28.752374] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:62, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2031742408, recv_hp_rpc_cnt:5231, recv_np_rpc_cnt:103800, recv_lp_rpc_cnt:0, recv_mysql_cnt:9369, recv_task_cnt:3750, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10822776, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=116291 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1841, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:93996, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736326288491094}{group_id:10, queue_size:0, recv_req_cnt:11194, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2087, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326228853642}{group_id:2, queue_size:0, recv_req_cnt:30106, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11896, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:4976, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:26775, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=271 pcode=0x717:cnt=179 pcode=0x1502:cnt=178 pcode=0x710:cnt=60 pcode=0x150b:cnt=60, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:51:28.753468] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1092] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:15279924, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8203, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3581183, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=17, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:51:28.754246] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=776] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:14942782, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3567770, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:51:28.754884] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=637] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:23, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:879619846, recv_hp_rpc_cnt:3737, recv_np_rpc_cnt:37995, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:779551, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=65603 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:67953, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:6, nesting_worker_cnt:0, token_change:1736326288491138}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2190, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326279406832}{group_id:2, queue_size:0, recv_req_cnt:15501, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18582, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=296 pcode=0x51c:cnt=242 pcode=0x717:cnt=167 pcode=0x710:cnt=92 pcode=0x150b:cnt=60, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:51:28.755560] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=675] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:60, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9595974173, recv_hp_rpc_cnt:77, recv_np_rpc_cnt:34984, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2948419, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=25070 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:0, recv_req_cnt:7026, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326269316514}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:150471, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:8, nesting_worker_cnt:0, token_change:1736326288607248}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326242277705}{group_id:19, queue_size:0, recv_req_cnt:408, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326264650105}{group_id:2, queue_size:0, recv_req_cnt:37176, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:11115, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=346 pcode=0x1501:cnt=171 pcode=0x717:cnt=170 pcode=0x1516:cnt=155 pcode=0x150b:cnt=117, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:51:28.761510] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AF-0-0] [lt=176] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:28.761557] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2AF-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326288318549003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326288318549003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:28.764722] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=24] clear dtl interm result cost(us)(clear_cost=3933, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:28.765377] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43341154}, readable_log_tail:{lsn:43341154}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23763614, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:28.765468] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=89] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:28.765502] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:28.770693] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=51] clear dtl interm result cost(us)(clear_cost=13, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:28.786497] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=107] BlockGCTimerTask success(ret=0, cost_time_us=16, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:28.802432] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:28.802790] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=40] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1003}, {id:1}]) [2025-01-08 00:51:28.803098] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:28.803136] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:28.803301] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=31] gc stale ls task succ [2025-01-08 00:51:28.807480] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:28.807592] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=111][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:28.808022] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58538][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8717-0-0] [lt=85] init point thread id with(&point=0x5572d50e53c0, idx_=1194, point=[thread id=58538, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58538) [2025-01-08 00:51:28.813035] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3D7-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:28.818400] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870E2-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:28.818496] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:28.818512] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:28.821527] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.826905] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D071-0-0] [lt=23][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:28.827567] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=9] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:28.828593] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=13] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=651907072, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=421834752, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:28.828804] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=117] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:28.828888] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=81] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2848, drift_size_=0, pre_drift_size=-6291456, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:28.828953] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:28.829063] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:28.829082] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:28.830409] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58545][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:28.830548] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58545][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A2A-0-0] [lt=51] init point thread id with(&point=0x5572d50e5580, idx_=1201, point=[thread id=58545, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58545) [2025-01-08 00:51:28.830800] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185433-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:28.830981] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:28.830988] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=996] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:51:28.831022] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=34] temporary file module use no blocks [2025-01-08 00:51:28.831034] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:28.831047] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:28.831060] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193792742956, flush_io_finished_ret_:0, flush_io_finished_round_:1947, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:28.831104] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:28.831119] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-01-08 00:51:28.831125] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:28.831132] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:28.831141] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193792742969, flush_io_finished_ret_:0, flush_io_finished_round_:3775, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:28.842347] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:28.843459] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=12] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:28.843746] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:28.847508] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:28.847586] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:28.852842] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=99] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:28.856427] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:28.861600] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-01-08 00:51:28.862875] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58033][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=28] worker thread exit(tenant_->id()=1, workers_.get_size()=1) [2025-01-08 00:51:28.878859] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1736326288878856, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006369, add_bytes: 0 [2025-01-08 00:51:28.896548] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.896617] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.896659] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.897528] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:28.904824] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:332K/76:148766 disk:332K/76:148766 t1m2:40K/9:173505 t1m2g0:40K/9:173505 t1002m2:216K/50:90401 t1002m2g0:216K/50:90401 t1001m2:76K/17:307332 t1001m2g0:76K/17:307332 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:28.910268] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2BE-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326288924807, keepalive_ts:-1}) [2025-01-08 00:51:28.910298] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2BE-0-0] [lt=28] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326288924807, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:28.911446] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:28.912359] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:28.913991] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=943710, clean_num=31457) [2025-01-08 00:51:28.915964] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:28.916677] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2BF-0-0] [lt=35] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326288925027, keepalive_ts:-1}) [2025-01-08 00:51:28.916718] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2BF-0-0] [lt=39] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326288925027, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:28.916908] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDF-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:28.916934] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CDF-0-0] [lt=19] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=28) [2025-01-08 00:51:28.919185] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58523][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2E-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326288933886, keepalive_ts:-1}) [2025-01-08 00:51:28.919274] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58523][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2E-0-0] [lt=87] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326288933886, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:28.921729] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:28.929058] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F7-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326288941569, keepalive_ts:-1}) [2025-01-08 00:51:28.929090] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F7-0-0] [lt=30] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326288941569, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:28.945168] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA7-0-0] [lt=35][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:28.946951] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA7-0-0] [lt=93][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:29.000098] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326284984691, last_first_register_time_us=1736324449183670, register_time_us=1736326284984691, curr_time_us=1736326289000095) [2025-01-08 00:51:29.000258] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=107] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326289000207, last_submit_register_req_time_us=1736326289000207, register_parent_reason=1) [2025-01-08 00:51:29.000277] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:29.002835] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BA9-0-0] [lt=103] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:29.004759] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=81, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:29.004844] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:29.004863] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:29.004878] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:29.004948] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=56, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:29.004966] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:29.004978] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:29.004988] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:29.006138] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:29.006273] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:29.007433] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC394F-0-0] [lt=14][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18113, log_task={header:{begin_lsn:{lsn:2204622}, end_lsn:{lsn:2204744}, log_id:18113, min_scn:{val:1736326288811615000, v:0}, max_scn:{val:1736326288811615000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2204500}, prev_proposal_id:1, committed_end_lsn:{lsn:2204622}, data_checksum:940009244, accum_checksum:491348872, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326288906999, freeze_ts:1736326288906999, submit_ts:1736326288907005, flushed_ts:1736326288908052, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:1047}, fs_cb_begin_ts=1736326289007407, log_life_time=100408) [2025-01-08 00:51:29.011437] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=211, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:29.011462] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=8, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:29.020312] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=146] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:29.022267] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.022525] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=2242, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:29.022548] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9640) [2025-01-08 00:51:29.022567] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=496, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:29.022593] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9640) [2025-01-08 00:51:29.022611] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=13, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:29.025472] INFO [LIB] stat (utility.h:1210) [58523][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A2D-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=11, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:29.025506] INFO [LIB] stat (utility.h:1210) [58523][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A2D-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=4350, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:29.025520] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58523][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A2D-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=104, avg_log_submit_to_flush_time=2767, avg_log_submit_to_slide_time=4246) [2025-01-08 00:51:29.025745] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9640, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:29.025936] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326288308139003, v:0}, new_max_version={val:1736326288308139003, v:0}, last_min_version={val:1736326288308139002, v:0}, last_max_version={val:1736326288308139002, v:0}, total_time_us=8013, delta_us=717795) [2025-01-08 00:51:29.025969] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326288308139003, v:0}, min_version={val:1736326288308139003, v:0}, max_version={val:1736326288308139003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:29.028022] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:29.067576] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=38] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:29.078230] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5263569836, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:29.078287] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:29.078303] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a1eb0) [2025-01-08 00:51:29.078312] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5263569836, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:29.107869] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9355-0-0] [lt=125] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:29.107911] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9355-0-0] [lt=41] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4736, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:29.110672] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3951-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:29.110718] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3951-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:29.110737] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58520][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3951-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:29.111949] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=77] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:29.113277] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8722-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:29.113314] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8722-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:29.113325] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8722-0-0] [lt=11] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12476, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:29.113377] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8722-0-0] [lt=50] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=1314, avg_log_submit_to_slide_time=12462) [2025-01-08 00:51:29.113472] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:29.113506] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18126, miss_cnt:0, cache_read_size:2217509, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:29.113980] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1578, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:29.114004] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:29.114020] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=99, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:29.114125] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=48, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:29.114521] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:29.114625] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF5A-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326289227820, keepalive_ts:-1}) [2025-01-08 00:51:29.114650] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF5A-0-0] [lt=24] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326289227820, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:29.121701] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=9762, avg_group_log_size=542) [2025-01-08 00:51:29.122544] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.150527] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C674-0-0] [lt=29][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193794064358, cache_obj->added_lc()=false, cache_obj->get_object_id()=13641, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:29.151859] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=1911, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:29.151962] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=78] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=237, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:29.152031] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=56, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:29.171029] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=30] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:29.171762] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=91] [COMPACTION DAG STATUS] (start_time=1736326279110161, end_time=1736326289171760, dag_status={}) [2025-01-08 00:51:29.171795] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=30] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:29.171809] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:29.171817] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:29.171824] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:29.171831] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:29.171838] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:29.171845] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:29.171852] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:29.171859] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:29.171866] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.171878] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.171886] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.171905] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.171914] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.171923] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.171930] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.171938] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172036] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=96] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172050] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172060] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172068] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172085] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172093] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172101] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172109] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172117] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172126] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172134] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172142] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172150] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172158] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172166] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172239] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172252] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172261] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172269] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172277] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172285] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172293] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172301] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172309] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172317] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172332] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172340] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172348] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172356] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172364] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172372] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172381] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172389] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172397] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172413] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172421] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172428] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172487] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172496] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172504] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172512] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172520] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172528] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172536] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172544] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172551] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172559] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172567] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172575] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172583] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172591] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172651] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172661] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172669] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172677] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172685] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172693] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172701] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172709] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172716] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172724] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172732] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:29.172742] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:29.172750] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:29.172758] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:29.172763] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:29.172769] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:29.172774] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:29.172779] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:29.172784] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:29.172790] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:29.172794] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:29.172801] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:29.190569] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=21] ====== check clog disk timer task ====== [2025-01-08 00:51:29.190603] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=28] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:29.206698] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1413, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:29.206805] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=105] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:29.206828] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=344, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:29.206864] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:29.207355] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1405, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:29.207379] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:29.207393] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=121, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:29.207417] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:29.211033] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9357-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=34, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:29.211054] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9357-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4800, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:29.211161] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:29.211867] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46B3-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:29.211897] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46B3-0-0] [lt=22] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:29.213003] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1126, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:29.213026] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:29.213086] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=55] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=113, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:29.213139] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=70, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:29.220801] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=99, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:29.221038] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=212] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9639) [2025-01-08 00:51:29.222773] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.278303] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:29.283252] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=27] ====== check clog disk timer task ====== [2025-01-08 00:51:29.283293] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=35] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:29.309854] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:29.310149] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=492, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:29.312132] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:29.312701] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A935A-0-0] [lt=45] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=14, avg_log_gen_to_submit_time=128, avg_log_submit_to_flush_time=1854, avg_log_submit_to_slide_time=4935) [2025-01-08 00:51:29.312842] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:29.315020] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:29.319392] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.321458] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.323065] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.333453] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:29.333489] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-01-08 00:51:29.333499] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:29.333513] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:29.333527] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193793246326, flush_io_finished_ret_:0, flush_io_finished_round_:1952, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:29.337167] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D44-0-0] [lt=38] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:29.338892] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=27] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-01-08 00:51:29.342619] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=32] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:29.343895] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=38] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:29.347680] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=183] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:29.347838] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:29.353033] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:29.356639] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=68631552) [2025-01-08 00:51:29.376404] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:29.376439] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=31] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:29.376452] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:29.376463] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:29.379116] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:29.379947] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=827] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:29.381473] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58546][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:29.381529] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC1-0-0] [lt=24] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:29.381553] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC1-0-0] [lt=16] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:29.381685] INFO [SERVER.OMT] recv_group_request (ob_tenant.cpp:1413) [46143][pnio1][T0][YB42C0A89F8B-00062B2D187BAEC1-0-0] [lt=37] worker thread created(id()=1, group->group_id_=19) [2025-01-08 00:51:29.382397] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1736326289382394, bytes: 877305319, bw: 0.266399 MB/s, add_ts: 1004605, add_bytes: 280626 [2025-01-08 00:51:29.387137] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBF-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:29.387268] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CBF-0-0] [lt=127] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:29.400648] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7] LogLoopThread round_cost_time(us)(round_cost_time=25) [2025-01-08 00:51:29.404027] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB63-0-0] [lt=45] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:29.404063] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB63-0-0] [lt=27] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:29.407487] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2188213, this={dio_aligned_buf:{buf_write_offset:1071, buf_padding_size:3025, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:22823, truncate_used_ts:5118}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2187264, buf_len=122, total_write_size=10473, total_write_size_after_dio=360448, ob_pwrite_used_ts=1556512, count=86, each_pwrite_cost=18098) [2025-01-08 00:51:29.418641] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326288807055003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=611575, in_cluster_service=true, cluster_version={val:1736326288807055000, v:0}, min_cluster_version={val:1736326288807055000, v:0}, max_cluster_version={val:1736326288807055000, v:0}, get_cluster_version_err=0, cluster_version_delta=611575, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18428, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=517336157, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:29.422065] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21] LogLoopThread round_cost_time(us)(round_cost_time=15) [2025-01-08 00:51:29.422261] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=34] ====== tenant freeze timer task ====== [2025-01-08 00:51:29.422163] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:44797, max_lsn:84354788, max_scn:{val:1736326289225398000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84354788}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42411748}}, last_submit_lsn:{lsn:84354203}, last_submit_end_lsn:{lsn:84354788}, last_submit_log_id:44797, last_submit_log_pid:1, max_flushed_lsn:{lsn:84354203}, max_flushed_end_lsn:{lsn:84354788}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84354788}, last_slide_log_id:44797, last_slide_scn:{val:1736326289225398000, v:0}, last_slide_lsn:{lsn:84354203}, last_slide_end_lsn:{lsn:84354788}, last_slide_log_pid:1, last_slide_log_accum_checksum:3252457152, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:44798, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84354788}, readable_log_tail:{lsn:84354788}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49854748, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:51:29.422455] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=286] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326289228847, last_flushed_end_lsn:{lsn:84354788}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326289235886, last_flushed_end_lsn:{lsn:84354788}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326289230355, last_flushed_end_lsn:{lsn:84354788}}]) [2025-01-08 00:51:29.422479] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=44798, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326289228847, last_flushed_end_lsn:{lsn:84354788}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326289235886, last_flushed_end_lsn:{lsn:84354788}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326289230355, last_flushed_end_lsn:{lsn:84354788}}]) [2025-01-08 00:51:29.423479] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.428605] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3417) [2025-01-08 00:51:29.428691] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=82] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:29.428904] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:29.428926] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71124160, tenant_id=1001) [2025-01-08 00:51:29.428947] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:29.428973] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] TxData Memory Statistic(logstream info): (ls_id={id:1}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=19) [2025-01-08 00:51:29.428989] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:595) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] TxData Memory Statistic : (Tenant Total Memory(MB)=1024, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=19, Freeze TxData Trigger Memory(MB)=20, Total TxDataTable Hold Memory(MB)=22, Total TxDataTable Memory Limit(MB)=204) [2025-01-08 00:51:29.430110] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3416) [2025-01-08 00:51:29.430213] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3417) [2025-01-08 00:51:29.431294] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C48-0-0] [lt=75] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=23, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:29.431319] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C48-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=3484, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:29.431330] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C48-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=95, avg_log_submit_to_flush_time=2540, avg_log_submit_to_slide_time=3389) [2025-01-08 00:51:29.431506] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3416, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:29.457281] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=111] gc stale ls task succ [2025-01-08 00:51:29.458746] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.458785] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.463719] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB606-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:29.463830] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB606-0-0] [lt=89] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326289313850000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326289313850000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:29.476328] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB606-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326289476290000, start_id=1736326289476290000, end_id=1736326289476290001) [2025-01-08 00:51:29.480268] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=29] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:29.496445] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:29.504300] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58478][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=45] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:51:29.505287] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=98] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8709, offset=2472587}, end_cursor:ObLogCursor{file_id=1, log_id=8710, offset=2472892}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b08050, len:1397, log_data_len:305, seq:8709, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:29.507291] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5D-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:29.507359] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5D-0-0] [lt=47] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326289309715000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326289309715000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:29.511000] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=45] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:29.512375] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:29.519674] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.519744] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:29.521049] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D8C6-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=26, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:29.521075] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D8C6-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25144, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:29.521353] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D8C6-0-0] [lt=228] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=8, avg_log_submit_to_flush_time=1495, avg_log_submit_to_slide_time=25135) [2025-01-08 00:51:29.521771] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:29.521808] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18029, miss_cnt:0, cache_read_size:2205467, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:29.522304] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=74] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:29.522329] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] try gc retain ctx [2025-01-08 00:51:29.522337] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-01-08 00:51:29.522360] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1736326289419518000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43347020}, palf_committed_end_scn_:{val:1736326289476290000, v:0}, last_check_scn_:{val:1736326289514488000, v:0}, max_applied_cb_scn_:{val:1736326289419518000, v:0}}) [2025-01-08 00:51:29.522385] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] finish update min start scn(max_decided_scn={val:1736326289419518000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326279403283001, v:0}, keep_alive_scn_:{val:1736326284383131001, v:0}, update_ts_:1736326289521396}) [2025-01-08 00:51:29.523166] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299284252000, latest_log_ts:{val:1736326282955407001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:29.523686] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.526432] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=97] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8710, offset=2472892}, end_cursor:ObLogCursor{file_id=1, log_id=8711, offset=2473189}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b0c050, len:1092, log_data_len:297, seq:8710, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:29.526521] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:1736326289427266000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84355373}, palf_committed_end_scn_:{val:1736326289427266000, v:0}, last_check_scn_:{val:1736326289427266000, v:0}, max_applied_cb_scn_:{val:1736326289427266000, v:0}}) [2025-01-08 00:51:29.526549] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84355373}, readable_log_tail:{lsn:84355373}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49854163, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:29.527815] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5D-0-0] [lt=100] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:29.527870] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5D-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326289309715001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326289309715001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:29.539135] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8711, offset=2473189}, end_cursor:ObLogCursor{file_id=1, log_id=8712, offset=2473486}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b10050, len:795, log_data_len:297, seq:8711, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:29.540240] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=44] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8712, offset=2473486}, end_cursor:ObLogCursor{file_id=1, log_id=8713, offset=2473783}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b14050, len:498, log_data_len:297, seq:8712, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:29.542770] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.543714] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.551316] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:29.551364] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=46] try gc retain ctx [2025-01-08 00:51:29.551372] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:29.551413] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326289427266000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326279315890001, v:0}, keep_alive_scn_:{val:1736326284359372001, v:0}, update_ts_:1736326289550643}) [2025-01-08 00:51:29.552325] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAECB-0-0] [lt=45] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:29.552365] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAECB-0-0] [lt=29] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:29.552467] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299254996000, latest_log_ts:{val:1736326282603912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:29.552773] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=166] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=74, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:29.552804] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=3407, avg_group_log_size=378) [2025-01-08 00:51:29.574245] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:51:29.615076] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:29.616624] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58473][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=32] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:29.621648] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ADC7-0-0] [lt=67] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.840151977539062500e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=392, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.431660989222258422e+02) [2025-01-08 00:51:29.630627] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.630957] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BAB-0-0] [lt=17] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:29.633074] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=68] get number(ret=0, service_type_=0, range=1, base_id=1736326289632138000, start_id=1736326289632138000, end_id=1736326289632138001) [2025-01-08 00:51:29.672934] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BECE-0-0] [lt=12] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:29.696284] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=29] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:29.705766] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:29.705877] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, bw:40845, [delay/us]:prepare:7, schedule:43, submit:103, rt:1292, total:1447) [2025-01-08 00:51:29.705898] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:29.706010] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=96] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40847, [delay/us]: prepare:7, schedule:43, submit:103, device:1292, total:1447) [2025-01-08 00:51:29.706026] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:29.706113] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:18, bw:81694, [delay/us]:prepare:6, schedule:197, submit:243, rt:1550, total:1999) [2025-01-08 00:51:29.706128] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:29.706168] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:18, avg_bw:81690, [delay/us]: prepare:6, schedule:197, submit:243, device:1550, total:1999) [2025-01-08 00:51:29.706253] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=79] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:29.706279] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16336, [delay/us]:prepare:7, schedule:33, submit:1, rt:1021, total:1062) [2025-01-08 00:51:29.706302] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:163368, [delay/us]:prepare:4, schedule:65, submit:221, rt:1126, total:1418) [2025-01-08 00:51:29.706311] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:29.706343] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16336, [delay/us]: prepare:7, schedule:33, submit:1, device:1021, total:1062) [2025-01-08 00:51:29.706411] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:163369, [delay/us]: prepare:4, schedule:65, submit:221, device:1126, total:1418) [2025-01-08 00:51:29.706424] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:29.706444] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:382, total:394) [2025-01-08 00:51:29.706470] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:29.706554] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=79] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4084, [delay/us]: prepare:0, schedule:0, submit:600000000, device:382, total:394) [2025-01-08 00:51:29.706573] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:29.706590] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:29.706598] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:29.706607] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:29.706615] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=279kB/s, net_out=221kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:29.709027] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=25] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:29.711774] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=85] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:29.712559] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=28] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:29.712603] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=44] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:29.712789] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84988-0-0] [lt=17] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:29.712830] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84988-0-0] [lt=37] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:29.712899] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:29.720345] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=146] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:29.731113] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.754369] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=72] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:29.756426] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1736326289755705000, start_id=1736326289755705000, end_id=1736326289755705001) [2025-01-08 00:51:29.759216] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18699869, this={dio_aligned_buf:{buf_write_offset:2174, buf_padding_size:1922, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:32614, truncate_used_ts:6730}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=18698240, buf_len=545, total_write_size=13830, total_write_size_after_dio=368640, ob_pwrite_used_ts=1588056, count=87, each_pwrite_cost=18253) [2025-01-08 00:51:29.769376] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:29.770747] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=42] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:29.770864] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=42] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326284334180000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:29.770934] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=66] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326284334180000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326284334180000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:29.772173] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=26] success to update min reserved snapshot(reserved_snapshot=1736324485633619000, duration=1800, snapshot_gc_ts=1736326285633619000) [2025-01-08 00:51:29.774825] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=84] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324485633619000) [2025-01-08 00:51:29.778034] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] submit reserved snapshot log success(ls_id={id:1001}, new_snapshot_version=1736324485633619000) [2025-01-08 00:51:29.778116] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=76] update reserved snapshot finished(cost_ts=5946, reserved_snapshot=1736324485633619000) [2025-01-08 00:51:29.785782] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B0-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:29.785848] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B0-0-0] [lt=47] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326289322864003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326289322864003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:29.796514] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:29.797838] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:29.797928] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326285633619000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:29.798506] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=44] ====== check clog disk timer task ====== [2025-01-08 00:51:29.798530] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:29.802645] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=44] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:29.803416] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=88] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1003}, {id:1002}, {id:1}]) [2025-01-08 00:51:29.803484] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:29.803512] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:29.806333] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=18] gc stale ls task succ [2025-01-08 00:51:29.807008] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3DA-0-0] [lt=40][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:29.807646] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=8] finish to resize log external storage handler(current_ts=1736326289807641, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:29.807824] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=176] snapshot_traffic_info once [2025-01-08 00:51:29.809560] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:29.809581] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:29.813172] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=31] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:29.815979] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870E8-0-0] [lt=60][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:29.816034] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:29.816047] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:29.819632] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58542][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A3B-0-0] [lt=183] init point thread id with(&point=0x5572d50e54c0, idx_=1198, point=[thread id=58542, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58542) [2025-01-08 00:51:29.822619] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2218241}], scn_array=[cnt:1, {val:1736326289322864004, v:0}], curr_size=122, accum_size=732, time_cost=10903) [2025-01-08 00:51:29.822938] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=24] ====== checkpoint timer task ====== [2025-01-08 00:51:29.823684] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ADCA-0-0] [lt=101] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=2, total_release_part_ctx_count=105) [2025-01-08 00:51:29.823913] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=24] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:2, active_tx_count:2, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:29.823959] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=45] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1736326289811506000, v:0}) [2025-01-08 00:51:29.824083] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299284252000, latest_log_ts:{val:1736326282955407001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:29.824170] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=83] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:29.824270] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=90] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:29.824286] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1736326282955407001, v:0}) [2025-01-08 00:51:29.824295] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:29.824303] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:29.824315] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:29.824333] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:29.824347] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:29.824377] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:29.824400] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:29.824410] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:29.826494] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D074-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:29.828286] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43350914}], scn_array=[cnt:1, {val:1736326289811506000, v:0}], curr_size=750, accum_size=7871, time_cost=15339) [2025-01-08 00:51:29.832054] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=122] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326289621718000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=210327, in_cluster_service=true, cluster_version={val:1736326289528649000, v:0}, min_cluster_version={val:1736326289528649000, v:0}, max_cluster_version={val:1736326289528649000, v:0}, get_cluster_version_err=0, cluster_version_delta=303396, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18868, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=635468828, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:29.832951] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185434-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:29.833246] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.839894] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=58] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:29.841157] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=42] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:29.841343] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=48] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326285742394000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:29.841391] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=45] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326285742394000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326285742394000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:29.842837] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:29.844386] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=50] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:29.844479] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:29.844733] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43351664}], scn_array=[cnt:1, {val:1736326289821344000, v:0}], curr_size=589, accum_size=8460, time_cost=16163) [2025-01-08 00:51:29.846095] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58549][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:29.846145] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77] worker thread created(tenant_->id()=1001, group_id_=1, token=8) [2025-01-08 00:51:29.847993] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:29.848346] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:29.853211] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:29.856908] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:29.862622] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=22] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:29.864991] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=49] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:29.866842] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326289865971000, start_id=1736326289865971000, end_id=1736326289865971001) [2025-01-08 00:51:29.867933] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857CC-0-0] [lt=9] MediumCheckTask(cost_ts=9) [2025-01-08 00:51:29.886882] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=91] PNIO [ratelimit] time: 1736326289886878, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008022, add_bytes: 0 [2025-01-08 00:51:29.889020] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=89] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:29.890501] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326289890308000, start_id=1736326289890308000, end_id=1736326289890308001) [2025-01-08 00:51:29.893538] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A3D-0-0] [lt=185] init point thread id with(&point=0x5572d50e5680, idx_=1205, point=[thread id=58549, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58549) [2025-01-08 00:51:29.895099] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=35] [APPLY STAT CB APPEND COST TIME](cur_stat_count=53, stat_interval=5000000, avg (count/cost)=148, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:29.895140] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=40] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=53, stat_interval=5000000, avg (count/cost)=7343, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:29.895152] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=53, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:29.895160] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=53, stat_interval=5000000, avg (count/cost)=-1006, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:29.895167] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=53, stat_interval=5000000, avg (count/cost)=6485, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:29.896822] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.897319] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.897379] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.897786] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:29.906056] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=27] ====== checkpoint timer task ====== [2025-01-08 00:51:29.906829] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:29.906872] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=42] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:29.906968] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299254996000, latest_log_ts:{val:1736326282603912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:29.907048] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=76] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299254996000, latest_log_ts:{val:1736326282603912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:29.907104] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=50] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:29.907170] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=63] get rec log scn(service_type_=0, rec_log_ts={val:1736326282603912001, v:0}) [2025-01-08 00:51:29.907240] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=69] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:29.907248] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:29.907262] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:29.907281] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:29.907294] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:29.907313] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:29.907334] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:29.907344] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:29.910815] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=31] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:320K/76:126588 disk:320K/76:126588 t1m2:44K/10:61156 t1m2g0:44K/10:61156 t1002m2:188K/47:80559 t1002m2g0:188K/47:80559 t1001m2:88K/19:274889 t1001m2g0:88K/19:274889 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:29.913106] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:29.920786] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:29.928121] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:29.933897] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AA4-0-0] [lt=41] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:29.933954] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:29.956096] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA8-0-0] [lt=53][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:29.957376] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=45] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12805025792, fast_delete_log_mode=false) [2025-01-08 00:51:29.959651] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA8-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:29.964982] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46383][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C679-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193794878816, cache_obj->added_lc()=false, cache_obj->get_object_id()=13642, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:29.969441] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=71] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:29.974940] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C67E-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1736326289974034000, start_id=1736326289974034000, end_id=1736326289974034001) [2025-01-08 00:51:29.978103] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=24] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:30.007787] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=48] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326285986296, last_first_register_time_us=1736324454612308, register_time_us=1736326285986296, curr_time_us=1736326290007784) [2025-01-08 00:51:30.007974] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=145] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326290007935, last_submit_register_req_time_us=1736326290007935, register_parent_reason=1) [2025-01-08 00:51:30.007993] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:30.010649] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=87, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:30.010682] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1633, avg_group_log_size=163) [2025-01-08 00:51:30.010701] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1633) [2025-01-08 00:51:30.010716] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:30.010766] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=62, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:30.010798] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1210, avg_group_log_size=121) [2025-01-08 00:51:30.010854] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1210) [2025-01-08 00:51:30.010866] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:30.011042] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [58548][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=0] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5265024267, tenant_id:500}, cur_thread_count=3, cnt=1, new_thread_count=4) [2025-01-08 00:51:30.012003] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] succ to submit log task to replay service(to_submit_lsn={lsn:2206011}, to_submit_scn={val:1736326289811506001, v:0}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326290011857, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2206077}, next_to_submit_scn_:{val:1736326289811506002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2205955}, end_lsn:{lsn:2206077}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2206077}, readable_log_tail:{lsn:2206077}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64898691, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326289811506001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326289811506001, v:0}, data_checksum:2124447694, flag:41128}}, init_mode_version:0, accumulate_checksum:2858291967, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326289811506001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18034, miss_cnt:0, cache_read_size:2206077, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:30.012284] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=156, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:30.012304] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:30.012764] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1633) [2025-01-08 00:51:30.014757] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [58548][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=42] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5265024267, tenant_id:500}, cur_thread_count=3, cnt=1, new_thread_count=4) [2025-01-08 00:51:30.016525] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9371-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1210) [2025-01-08 00:51:30.024535] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:30.025931] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=76] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=3548, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:30.025958] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10184) [2025-01-08 00:51:30.025976] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=654, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:30.026003] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=7] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10184) [2025-01-08 00:51:30.026020] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=33, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:30.027552] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DA7-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=12, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:30.027574] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DA7-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=6112, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:30.027585] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DA7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=413, avg_log_gen_to_submit_time=528, avg_log_submit_to_flush_time=4239, avg_log_submit_to_slide_time=5584) [2025-01-08 00:51:30.027741] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10184, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:30.027991] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326289309715004, v:0}, new_max_version={val:1736326289309715004, v:0}, last_min_version={val:1736326289309715002, v:0}, last_max_version={val:1736326289309715002, v:0}, total_time_us=6000, delta_us=718274) [2025-01-08 00:51:30.028040] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326289309715004, v:0}, min_version={val:1736326289309715004, v:0}, max_version={val:1736326289309715004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:30.028063] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:30.034169] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.114573] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=77] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1336880, replace_num=15728) [2025-01-08 00:51:30.114623] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:30.114692] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=139] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=36, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:30.115340] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF5B-0-0] [lt=33] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326290230018, keepalive_ts:-1}) [2025-01-08 00:51:30.115373] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF5B-0-0] [lt=32] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326290230018, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:30.115774] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC873F-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:30.115795] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC873F-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4820, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:30.116438] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC873E-0-0] [lt=38] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:30.116470] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC873E-0-0] [lt=23] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:30.116481] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC873E-0-0] [lt=10] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14220, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:30.116490] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC873E-0-0] [lt=6] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=63, avg_log_submit_to_flush_time=3056, avg_log_submit_to_slide_time=14156) [2025-01-08 00:51:30.116550] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:30.116580] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18136, miss_cnt:0, cache_read_size:2218729, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:30.121448] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=17] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:30.127462] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC6-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326290241881, keepalive_ts:-1}) [2025-01-08 00:51:30.127492] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC6-0-0] [lt=29] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326290241881, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:30.131343] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=240] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=10184, avg_group_log_size=565) [2025-01-08 00:51:30.134381] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.153708] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1302, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:30.153766] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=175, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:30.153827] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:30.168626] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C67F-0-0] [lt=29][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193795082456, cache_obj->added_lc()=false, cache_obj->get_object_id()=13643, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:30.172987] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3782) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-01-08 00:51:30.173018] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3805) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=29] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-01-08 00:51:30.181028] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A60-0-0] [lt=38] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:30.208463] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3967-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:30.208532] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3967-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1333, avg_group_log_size=121) [2025-01-08 00:51:30.208555] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3967-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:51:30.208598] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3967-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18125, log_task={header:{begin_lsn:{lsn:2206077}, end_lsn:{lsn:2206199}, log_id:18125, min_scn:{val:1736326289811506002, v:0}, max_scn:{val:1736326289811506002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2205955}, prev_proposal_id:1, committed_end_lsn:{lsn:2206077}, data_checksum:940009244, accum_checksum:1049451482, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326290107937, freeze_ts:1736326290107937, submit_ts:1736326290107950, flushed_ts:1736326290114656, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:6706}, fs_cb_begin_ts=1736326290208572, log_life_time=100635) [2025-01-08 00:51:30.210741] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2432, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:30.210772] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1333) [2025-01-08 00:51:30.210793] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=145, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:30.212775] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2946, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:30.212804] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1210) [2025-01-08 00:51:30.212820] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=250, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:30.212849] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=27, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:30.212966] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2230, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:30.213017] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1633) [2025-01-08 00:51:30.213035] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=215, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:30.213059] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2983, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:30.213071] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:30.217088] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A61-0-0] [lt=29] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:30.217405] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8742-0-0] [lt=34] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:30.217443] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8742-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5549, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:30.217554] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1210, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:30.223838] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get number(ret=0, service_type_=0, range=1, base_id=1736326290220960000, start_id=1736326290220960000, end_id=1736326290220960001) [2025-01-08 00:51:30.223988] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=74, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:30.224258] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=244] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10185) [2025-01-08 00:51:30.230280] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A62-0-0] [lt=47] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:30.234460] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A63-0-0] [lt=42] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:30.234703] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.241003] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A64-0-0] [lt=114] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:30.276817] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=17] ls blacklist refresh finish(cost_time=20877) [2025-01-08 00:51:30.278518] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:30.280574] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=20] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5265832406, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:30.280610] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:30.280624] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8db9c33e0) [2025-01-08 00:51:30.280633] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5265832406, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:30.304465] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A65-0-0] [lt=49] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:30.310512] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46C9-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:30.310624] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46C9-0-0] [lt=101] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:30.312036] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=20] ts response task statistics(alloc_count=60338, free_count=30168) [2025-01-08 00:51:30.312068] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:30.312930] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:30.312962] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=132, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:30.314007] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=46, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:30.315571] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:30.318032] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=104, avg_log_submit_to_flush_time=3004, avg_log_submit_to_slide_time=4875) [2025-01-08 00:51:30.318110] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1633, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:30.318956] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58552][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:30.319497] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.319517] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.319577] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:30.321744] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A66-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:30.321969] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:30.333678] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:30.333736] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=50] temporary file module use no blocks [2025-01-08 00:51:30.333749] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:30.333763] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:30.333778] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193794247399, flush_io_finished_ret_:0, flush_io_finished_round_:1953, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:30.336316] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A67-0-0] [lt=22] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:30.338405] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=35] failed to get result(ret=-4008) [2025-01-08 00:51:30.338543] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=20] check new jobs(ret=0, tenant_id=1001, is_oracle_tenant=false, job_infos=[{tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326185605925, start_date_:1736325283345722, last_date_:1736326183353587, this_date_:0, next_date_:1736327083345722, end_date_:64060560000000000, total_:6596354, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736410783345628, last_date_:0, this_date_:0, next_date_:1736410783345628, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:51:30.338671] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=126] check all tenants(ret=0) [2025-01-08 00:51:30.338766] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.343445] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A69-0-0] [lt=8] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:30.343709] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:30.344787] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=53] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:30.348280] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=106] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:30.348757] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=90] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:30.353460] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:30.357133] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:30.378944] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=43] current monitor number(seq_=-1) [2025-01-08 00:51:30.384796] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1736326290384793, bytes: 877551135, bw: 0.233867 MB/s, add_ts: 1002399, add_bytes: 245816 [2025-01-08 00:51:30.389299] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC0-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:30.389347] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC0-0-0] [lt=44] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:30.437437] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4396) [2025-01-08 00:51:30.437486] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:30.438989] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.439061] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4397) [2025-01-08 00:51:30.439325] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4396) [2025-01-08 00:51:30.440020] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C52-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:30.440040] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C52-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3839, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:30.440050] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C52-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=123, avg_log_submit_to_flush_time=1893, avg_log_submit_to_slide_time=3716) [2025-01-08 00:51:30.440160] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4397, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:30.441327] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E61-0-0] [lt=25] MediumCheckTask(cost_ts=9) [2025-01-08 00:51:30.457451] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=53] gc stale ls task succ [2025-01-08 00:51:30.458749] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.458811] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=65] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.495636] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB607-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:30.495686] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB607-0-0] [lt=31] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326290313040000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326290313040000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:30.503351] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:30.509273] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50623][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB607-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326290508382000, start_id=1736326290508382000, end_id=1736326290508382001) [2025-01-08 00:51:30.511985] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58358][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=25] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:51:30.515250] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:30.515839] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=76] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:30.519548] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.519727] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58552][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC874C-0-0] [lt=25] init point thread id with(&point=0x5572d50e5740, idx_=1208, point=[thread id=58552, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58552) [2025-01-08 00:51:30.522472] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:30.539113] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.541863] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.543385] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=30] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:30.544498] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=58] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.545719] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8713, offset=2473783}, end_cursor:ObLogCursor{file_id=1, log_id=8714, offset=2474088}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b18050, len:4297, log_data_len:305, seq:8713, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:30.546362] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5E-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:30.546428] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5E-0-0] [lt=50] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326290311284001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326290311284001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:30.552927] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=101, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:30.556596] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=98] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8714, offset=2474088}, end_cursor:ObLogCursor{file_id=1, log_id=8715, offset=2474385}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b1c050, len:3992, log_data_len:297, seq:8714, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:30.557191] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5E-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:30.557246] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5E-0-0] [lt=39] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326290311284001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326290311284001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:30.565399] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=80] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8715, offset=2474385}, end_cursor:ObLogCursor{file_id=1, log_id=8716, offset=2474682}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b28050, len:3695, log_data_len:297, seq:8715, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:30.566472] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=93] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8716, offset=2474682}, end_cursor:ObLogCursor{file_id=1, log_id=8717, offset=2474979}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b2c050, len:3398, log_data_len:297, seq:8716, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:30.566636] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18039, miss_cnt:0, cache_read_size:2206687, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:30.611117] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D8DB-0-0] [lt=37] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=35, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:30.611153] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D8DB-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29915, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:30.611165] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D8DB-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=15, avg_log_submit_to_flush_time=3108, avg_log_submit_to_slide_time=29900) [2025-01-08 00:51:30.611331] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:30.625041] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ADD4-0-0] [lt=72] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.840303325653076172e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=392, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.431090780126017989e+02) [2025-01-08 00:51:30.637893] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4992, avg_group_log_size=453) [2025-01-08 00:51:30.639276] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:30.639362] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.661216] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=22] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:30.661265] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=40] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:30.661282] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=89) [2025-01-08 00:51:30.678886] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D2248464B-0-0] [lt=41] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:51:30.692628] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [46340][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=26] XA scheduler heartbeat task statistics(avg_time=1) [2025-01-08 00:51:30.711700] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:30.711793] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44788, [delay/us]:prepare:8, schedule:101, submit:64, rt:1185, total:1360) [2025-01-08 00:51:30.711813] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:30.711870] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44793, [delay/us]: prepare:8, schedule:101, submit:64, device:1185, total:1360) [2025-01-08 00:51:30.711883] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:30.711923] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:81443, [delay/us]:prepare:8, schedule:293, submit:1268, rt:1485, total:3055) [2025-01-08 00:51:30.711933] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:30.711968] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:81447, [delay/us]: prepare:8, schedule:293, submit:1268, device:1485, total:3055) [2025-01-08 00:51:30.711979] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:30.712000] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20363, [delay/us]:prepare:5, schedule:89, submit:0, rt:1077, total:1173) [2025-01-08 00:51:30.712022] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:41, bw:166980, [delay/us]:prepare:6, schedule:166, submit:488, rt:2125, total:2786) [2025-01-08 00:51:30.712031] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:30.712058] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20363, [delay/us]: prepare:5, schedule:89, submit:0, device:1077, total:1173) [2025-01-08 00:51:30.712074] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:41, avg_bw:166990, [delay/us]: prepare:6, schedule:166, submit:488, device:2125, total:2786) [2025-01-08 00:51:30.712138] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=58] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:30.712161] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4072, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:382, total:392) [2025-01-08 00:51:30.712227] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:30.712262] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4072, [delay/us]: prepare:0, schedule:0, submit:600000000, device:382, total:392) [2025-01-08 00:51:30.712281] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:30.712297] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:30.712305] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:30.712313] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:30.712320] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=215kB/s, net_out=241kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:30.717550] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=91] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:30.723047] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:30.727551] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085161-0-0] [lt=18] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:30.734488] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=37] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:30.734574] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=82] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:30.734590] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:30.734601] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:30.737341] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:51:30.738142] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=797] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:30.738858] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] success to update min reserved snapshot(reserved_snapshot=1736324484334180000, duration=1800, snapshot_gc_ts=1736326284334180000) [2025-01-08 00:51:30.739409] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.743099] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=37] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324484334180000) [2025-01-08 00:51:30.743149] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=47] update reserved snapshot finished(cost_ts=4317, reserved_snapshot=1736324484334180000) [2025-01-08 00:51:30.744347] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=23] ====== tenant freeze timer task ====== [2025-01-08 00:51:30.756592] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=20] [APPLY STAT CB APPEND COST TIME](cur_stat_count=34, stat_interval=5000000, avg (count/cost)=142, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:30.756624] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=32] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=34, stat_interval=5000000, avg (count/cost)=4901, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:30.756633] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=34, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:30.756641] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=34, stat_interval=5000000, avg (count/cost)=-826, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:30.756648] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=34, stat_interval=5000000, avg (count/cost)=4217, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:30.756880] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:30.757074] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=184] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:30.757093] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:30.757101] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:30.757108] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:30.757116] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=100) [2025-01-08 00:51:30.773056] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=92] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:30.803259] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:30.804106] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1002}, {id:1003}]) [2025-01-08 00:51:30.804225] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:30.804264] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:30.806445] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=35] gc stale ls task succ [2025-01-08 00:51:30.808868] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3DD-0-0] [lt=121][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:30.809407] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:30.809431] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:30.811957] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D077-0-0] [lt=23][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:30.813314] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326290813292], gts:1736326290812143000, latest_srr:[mts=1736326290813292]}) [2025-01-08 00:51:30.813411] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=83] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326290813398], gts:1736326290812143000, latest_srr:[mts=1736326290813398]}) [2025-01-08 00:51:30.813876] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326290813858], gts:1736326290812143001, latest_srr:[mts=1736326290813858]}) [2025-01-08 00:51:30.818495] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870EE-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:30.818636] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:30.818655] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:30.830280] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=59] get_max_applied_scn(scn={val:1736326290311284003, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2185937}, palf_committed_end_scn_:{val:1736326290311284004, v:0}, last_check_scn_:{val:1736326290311284004, v:0}, max_applied_cb_scn_:{val:1736326290311284003, v:0}}) [2025-01-08 00:51:30.830327] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2185937}, readable_log_tail:{lsn:2185937}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64918831, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:30.834947] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185435-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:30.835355] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:30.835381] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-01-08 00:51:30.835391] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:30.835404] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:30.835418] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193794747127, flush_io_finished_ret_:0, flush_io_finished_round_:1949, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:30.835479] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=84] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:30.835503] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-01-08 00:51:30.835515] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:30.835531] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:30.835548] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193794747174, flush_io_finished_ret_:0, flush_io_finished_round_:3777, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:30.836842] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B1-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:30.836884] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B1-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326290323279003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326290323279003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:30.839538] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.843875] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=50] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:30.844547] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:30.844967] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=36] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:30.845274] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58474][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=20] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:51:30.845781] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:30.849463] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:30.849642] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:30.854449] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:30.857037] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=42] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:30.857533] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:30.888135] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:30.888327] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=184] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:30.888345] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=13] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:30.888368] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=21] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:30.891164] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=25] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:30.891305] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1736326290891301, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004423, add_bytes: 0 [2025-01-08 00:51:30.896489] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.896628] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.897641] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=93] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:30.899338] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=46] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:30.899380] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=40] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:30.900379] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.900895] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487C-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:30.900976] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487C-0-0] [lt=34] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:30.900994] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487C-0-0] [lt=14] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:30.904357] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=13] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:30.906361] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=91] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:30.907346] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:30.908826] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=38] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:30.908849] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=21] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:30.910518] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487C-0-0] [lt=9] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:30.910661] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487C-0-0] [lt=47] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:30.910698] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487C-0-0] [lt=35] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:30.913584] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=15] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:30.915540] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=42] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:30.917062] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=19] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:30.917104] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=40] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:30.917383] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=45] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18697528}, readable_log_tail:{lsn:18697528}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48407240, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:30.917495] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=110] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:30.917512] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2185937}, readable_log_tail:{lsn:2185937}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64918831, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:30.917711] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=198] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:30.917725] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_replay_process success(id={id:1002}, submitted_log_size=2219461, unsubmitted_log_size=2219461, replayed_log_size=2219461, unreplayed_log_size=0) [2025-01-08 00:51:30.917736] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] get_replay_process success(id={id:1003}, submitted_log_size=2207053, unsubmitted_log_size=2207053, replayed_log_size=2207053, unreplayed_log_size=0) [2025-01-08 00:51:30.917802] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:30.918306] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:694) [46222][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=11] dump dest keepalive data states, addr: "192.168.159.137:2882", last_write_ts_=1736326290709384, last_read_ts_=1736326290508382, in_black_=0 [2025-01-08 00:51:30.918335] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:694) [46222][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=17] dump dest keepalive data states, addr: "192.168.159.138:2882", last_write_ts_=1736326290508382, last_read_ts_=1736326290709384, in_black_=0 [2025-01-08 00:51:30.918431] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:694) [46222][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=6] dump dest keepalive data states, addr: "192.168.159.139:2882", last_write_ts_=1736326290709384, last_read_ts_=1736326290508382, in_black_=0 [2025-01-08 00:51:30.918555] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46D5-0-0] [lt=78][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18235, log_task={header:{begin_lsn:{lsn:2219461}, end_lsn:{lsn:2219574}, log_id:18235, min_scn:{val:1736326290323279004, v:0}, max_scn:{val:1736326290323279004, v:0}, data_len:57, proposal_id:1, prev_lsn:{lsn:2219339}, prev_proposal_id:1, committed_end_lsn:{lsn:2219461}, data_checksum:1291593746, accum_checksum:2935285626, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326290815840, freeze_ts:1736326290815840, submit_ts:1736326290815875, flushed_ts:1736326290820395, gen_to_freeze cost time:0, gen_to_submit cost time:35, submit_to_flush cost time:4520}, fs_cb_begin_ts=1736326290918388, log_life_time=102548) [2025-01-08 00:51:30.919344] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:284K/68:168671 disk:284K/68:168671 t1m2:36K/9:65351 t1m2g0:36K/9:65351 t1002m2:176K/43:184990 t1002m2g0:176K/43:184990 t1001m2:72K/16:182928 t1001m2g0:72K/16:182928 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:30.919763] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=150][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2207053}], scn_array=[cnt:1, {val:1736326290313040005, v:0}], curr_size=112, accum_size=844, time_cost=23382) [2025-01-08 00:51:30.919870] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1, this=0x2ad8f0fff7a8, extra_info="log_io_worker_num:4, cb_thread_pool_tg_id:344") [2025-01-08 00:51:30.919929] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=127] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:30.920635] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE0-0-0] [lt=23] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:30.920662] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE0-0-0] [lt=21] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=30) [2025-01-08 00:51:30.921522] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487C-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:30.921613] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487C-0-0] [lt=40] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:30.921630] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487C-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:30.921712] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=13] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:30.922342] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=626] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:30.922395] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=44] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:30.922406] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:30.922427] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487C-0-0] [lt=16] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:30.922446] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487C-0-0] [lt=19] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:30.922480] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=27] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:30.922490] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:30.922514] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487C-0-0] [lt=21] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:30.922530] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487C-0-0] [lt=16] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:30.922544] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=8] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:30.922571] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=25] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:30.922594] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487C-0-0] [lt=20] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:30.922614] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487C-0-0] [lt=19] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:30.923982] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:30.924025] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=39] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:30.924036] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=10] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:30.924075] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:30.924088] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:30.924097] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:30.924109] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:30.924124] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487C-0-0] [lt=14] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:30.925101] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2207109}, scn={val:1736326290313040005, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:30.925220] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=16] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:30.925144] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:27, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326290924196, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2207165}, next_to_submit_scn_:{val:1736326290313040006, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2207053}, end_lsn:{lsn:2207165}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2207165}, readable_log_tail:{lsn:2207165}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64897603, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326290313040005, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326290313040005, v:0}, data_checksum:3081108696, flag:50532}}, init_mode_version:0, accumulate_checksum:3958630051, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326290313040005, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18043, miss_cnt:0, cache_read_size:2207165, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2207109}, scn_:{val:1736326290313040005, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326290898445, first_handle_ts_:1736326290925143, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad960a04150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:51:30.926726] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=28] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=975167, clean_num=31457) [2025-01-08 00:51:30.928586] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=540] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324485633619000) [2025-01-08 00:51:30.928616] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2207109}, scn_:{val:1736326290313040005, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326290898445, first_handle_ts_:1736326290928407, replay_cost_:206, retry_cost_:-1, read_log_buf_:0x2ad960a04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326290924196, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2207165}, next_to_submit_scn_:{val:1736326290313040006, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2207053}, end_lsn:{lsn:2207165}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2207287}, readable_log_tail:{lsn:2207287}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64897481, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326290313040005, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326290313040005, v:0}, data_checksum:3081108696, flag:50532}}, init_mode_version:0, accumulate_checksum:3958630051, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326290313040005, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18043, miss_cnt:0, cache_read_size:2207165, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=13, ret=0) [2025-01-08 00:51:30.939672] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:30.962685] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C681-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193795876518, cache_obj->added_lc()=false, cache_obj->get_object_id()=13644, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:30.968408] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CA9-0-0] [lt=54][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:30.969751] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CA9-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:30.986360] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDF5-0-0] [lt=48][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193795900191, cache_obj->added_lc()=false, cache_obj->get_object_id()=13645, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:31.012610] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=58] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=544, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:31.012653] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=43] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=7, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:31.017688] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:31.017733] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:31.017832] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:31.017856] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:31.017919] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=42, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:31.017938] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:31.017951] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:31.017962] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:31.019543] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:31.021677] INFO [SERVER] refresh_avail_memory (ob_table_load_assigned_memory_manager.cpp:117) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] ObTableLoadAssignedMemoryManager::refresh_avail_memory(MTL_ID()=1002, avail_memory=161061273, chunk_count_=0, avail_sort_memory_=161061273, avail_memory_=161061273) [2025-01-08 00:51:31.021978] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A938F-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:31.023086] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB APPEND COST TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=100, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:31.023111] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=25] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=6829, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:31.023120] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:31.023128] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=-861, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:31.023135] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=6067, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:31.027900] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=109] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:31.032565] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2293, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:31.032587] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9506) [2025-01-08 00:51:31.032605] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=587, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:31.032631] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9506) [2025-01-08 00:51:31.032648] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=31, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:31.035510] INFO [LIB] stat (utility.h:1210) [58526][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DB7-0-0] [lt=45] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=15, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:31.035529] INFO [LIB] stat (utility.h:1210) [58526][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DB7-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4797, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:31.035538] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58526][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DB7-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=184, avg_log_gen_to_submit_time=271, avg_log_submit_to_flush_time=2905, avg_log_submit_to_slide_time=4525) [2025-01-08 00:51:31.035702] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9506, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:31.036090] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=93] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326290311284004, v:0}, new_max_version={val:1736326290311284004, v:0}, last_min_version={val:1736326290311284002, v:0}, last_max_version={val:1736326290311284002, v:0}, total_time_us=10155, delta_us=724804) [2025-01-08 00:51:31.036120] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326290311284004, v:0}, min_version={val:1736326290311284004, v:0}, max_version={val:1736326290311284004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:31.039900] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.120325] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=16] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:31.121877] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9392-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:31.121903] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9392-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6640, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:31.127438] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=168] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:31.136390] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=59] get number(ret=0, service_type_=0, range=1, base_id=1736326291135802000, start_id=1736326291135802000, end_id=1736326291135802001) [2025-01-08 00:51:31.136621] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=50] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9973, avg_group_log_size=623) [2025-01-08 00:51:31.140000] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.155895] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=1541, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:31.155927] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=57, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:31.156002] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:31.190893] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B4BBAC7-0-0] [lt=115] get number(ret=0, service_type_=0, range=1, base_id=1736326291184873000, start_id=1736326291184873000, end_id=1736326291184873001) [2025-01-08 00:51:31.191248] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C683-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193796105031, cache_obj->added_lc()=false, cache_obj->get_object_id()=13646, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:31.191846] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-01-08 00:51:31.191877] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:31.198082] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=157, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:31.198159] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=75] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=7267, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:31.198376] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=216] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:31.198387] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=11] [APPLY STAT CB EXECUTE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=-3968, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:31.198456] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=3456, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:31.209537] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=213] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=101, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:31.210519] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9395-0-0] [lt=109] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1454) [2025-01-08 00:51:31.211105] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46DB-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:31.213370] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8760-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:31.213444] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8760-0-0] [lt=73] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21442, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:31.213459] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8760-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=18, avg_log_submit_to_flush_time=3048, avg_log_submit_to_slide_time=21424) [2025-01-08 00:51:31.213565] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:31.213596] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18147, miss_cnt:0, cache_read_size:2220062, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:31.220270] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3473, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:31.220354] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=82] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:31.220387] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=290, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:31.220567] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=33, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:31.220589] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5449, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:31.220607] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=43, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:31.220638] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:31.228338] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=60, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:31.228410] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9384) [2025-01-08 00:51:31.231274] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:31.237171] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326291035967000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=201196, in_cluster_service=true, cluster_version={val:1736326290835459000, v:0}, min_cluster_version={val:1736326290835459000, v:0}, max_cluster_version={val:1736326290835459000, v:0}, get_cluster_version_err=0, cluster_version_delta=401704, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36618, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=545734150, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:31.239563] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:31.241359] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:31.241566] WDIAG read_multi_row (ob_table_access_helper.h:238) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:31.243824] INFO delete_row (ob_table_access_helper.h:339) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326291241585) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326291241585) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:31.243898] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:31.243953] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:51:31.244056] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=54] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:51:31.244170] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:31.244249] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=77][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.278684] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:31.281498] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5268800204, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:31.281673] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=55] recycle one thread(this=0x2ad8885abcf0, total=4, remain=3) [2025-01-08 00:51:31.281716] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=41] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5268800204, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:31.283631] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] ====== check clog disk timer task ====== [2025-01-08 00:51:31.283679] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=41] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:31.308221] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC397D-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:31.308302] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC397D-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=12, avg_log_batch_cnt=0, total_group_log_size=1454, avg_group_log_size=121) [2025-01-08 00:51:31.309542] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2211749, this={dio_aligned_buf:{buf_write_offset:31, buf_padding_size:4065, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:24127, truncate_used_ts:5479}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=8192, aligned_block_offset=2207744, buf_len=122, total_write_size=11440, total_write_size_after_dio=397312, ob_pwrite_used_ts=970618, count=94, each_pwrite_cost=10325) [2025-01-08 00:51:31.309623] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=4433, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:31.309637] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1454) [2025-01-08 00:51:31.309655] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=968, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:31.310719] INFO [LIB] stat (utility.h:1210) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46DD-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:31.310764] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58520][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46DD-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:31.311612] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2450, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:31.312653] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8B9-0-0] [lt=85] get number(ret=0, service_type_=0, range=1, base_id=1736326291312444000, start_id=1736326291312444000, end_id=1736326291312444001) [2025-01-08 00:51:31.314960] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=530, get_gts_with_stc_cnt=29, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=29, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:31.314992] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:31.315037] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=511, get_gts_with_stc_cnt=72, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=73, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:31.319510] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=104] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.319636] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.319962] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2993, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:31.320003] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1765) [2025-01-08 00:51:31.320024] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=526, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:31.320068] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=41, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:31.320518] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:31.322244] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9398-0-0] [lt=18] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=84, avg_log_submit_to_flush_time=3328, avg_log_submit_to_slide_time=6130) [2025-01-08 00:51:31.322384] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1643, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:31.323836] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1829F54E-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326291323165000, start_id=1736326291323165000, end_id=1736326291323165001) [2025-01-08 00:51:31.326155] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50915][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F54F-0-0] [lt=315] get number(ret=0, service_type_=0, range=1, base_id=1736326291324638000, start_id=1736326291324638000, end_id=1736326291324638001) [2025-01-08 00:51:31.328018] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C7BB60C-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326291326722000, start_id=1736326291326722000, end_id=1736326291326722001) [2025-01-08 00:51:31.328017] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=75] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:31.329560] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46508][T1_L0_G0][T1][YB42C0A89F8B-00062B2D170BBDBC-0-0] [lt=105] get number(ret=0, service_type_=0, range=1, base_id=1736326291328448000, start_id=1736326291328448000, end_id=1736326291328448001) [2025-01-08 00:51:31.331073] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:31.331393] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58520][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=14] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:31.332377] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1323) [2025-01-08 00:51:31.332443] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=510, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:31.332498] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=46, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:31.333998] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:31.334032] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-01-08 00:51:31.334043] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:31.334057] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:31.334071] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193795247650, flush_io_finished_ret_:0, flush_io_finished_round_:1954, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:31.344072] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=79] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:31.344128] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D45-0-0] [lt=44] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:31.344514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.346672] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:31.349603] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:31.350017] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=60] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=4159488) [2025-01-08 00:51:31.354696] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:31.357681] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:31.379268] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=44] current monitor number(seq_=-1) [2025-01-08 00:51:31.391629] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC1-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:31.391666] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC1-0-0] [lt=34] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:31.392695] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=44] PNIO [ratelimit] time: 1736326291392692, bytes: 877802816, bw: 0.238141 MB/s, add_ts: 1007899, add_bytes: 251681 [2025-01-08 00:51:31.411389] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2220240}, scn={val:1736326290824977005, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:31.412002] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=99] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324485633619000) [2025-01-08 00:51:31.412032] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2220240}, scn_:{val:1736326290824977005, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326291441955, first_handle_ts_:1736326291411987, replay_cost_:42, retry_cost_:-1, read_log_buf_:0x2ad96d204240, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326291410991, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2220296}, next_to_submit_scn_:{val:1736326290824977006, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2220184}, end_lsn:{lsn:2220296}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2220296}, readable_log_tail:{lsn:2220296}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64884472, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326290824977005, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326290824977005, v:0}, data_checksum:3043426670, flag:15139}}, init_mode_version:0, accumulate_checksum:2388239167, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326290824977005, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18149, miss_cnt:0, cache_read_size:2220296, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=3, ret=0) [2025-01-08 00:51:31.432300] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-01-08 00:51:31.434839] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10] ====== tenant freeze timer task ====== [2025-01-08 00:51:31.434868] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70914440, tenant_id=1001) [2025-01-08 00:51:31.434967] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:31.439702] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=102] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3963) [2025-01-08 00:51:31.439785] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=77] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:31.440858] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3963) [2025-01-08 00:51:31.441009] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3963) [2025-01-08 00:51:31.442356] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C5C-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:31.442378] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C5C-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4891, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:31.442431] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C5C-0-0] [lt=51] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=99, avg_log_submit_to_flush_time=2276, avg_log_submit_to_slide_time=4792) [2025-01-08 00:51:31.442572] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=89] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3963, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:31.445032] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.457653] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=28] gc stale ls task succ [2025-01-08 00:51:31.458378] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.458445] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.519499] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=39] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.520695] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:31.524185] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB608-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:31.524245] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB608-0-0] [lt=43] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326291312444001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326291312444001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:31.528676] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:31.542715] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.543221] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=21] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:31.544086] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.545261] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.555297] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=99] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:31.572964] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=42] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8717, offset=2474979}, end_cursor:ObLogCursor{file_id=1, log_id=8718, offset=2475284}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b24050, len:3101, log_data_len:305, seq:8717, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:31.573640] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5F-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:31.573690] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5F-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326291315024002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326291315024002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:31.579211] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:491) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E5F-0-0] [lt=66] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=0, MAX_GAP=900000000000, new_sync_scn={val:1736326290311284001, v:0}, new_replayable_scn={val:1736326290311284001, v:0}, new_readable_scn={val:1736326290311284001, v:0}, old_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326289309715001, replayable_scn:1736326289309715001, readable_scn:1736326289309715001, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}) [2025-01-08 00:51:31.579655] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E5F-0-0] [lt=36] get number(ret=0, service_type_=0, range=1, base_id=1736326291578779000, start_id=1736326291578779000, end_id=1736326291578779001) [2025-01-08 00:51:31.584875] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=75] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8718, offset=2475284}, end_cursor:ObLogCursor{file_id=1, log_id=8719, offset=2475581}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b20050, len:2796, log_data_len:297, seq:8718, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:31.585370] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5F-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:31.585414] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E5F-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326291315024002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326291315024002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:31.592417] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=113] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8719, offset=2475581}, end_cursor:ObLogCursor{file_id=1, log_id=8720, offset=2475878}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b30050, len:2499, log_data_len:297, seq:8719, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:31.593570] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=52] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8720, offset=2475878}, end_cursor:ObLogCursor{file_id=1, log_id=8721, offset=2476175}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b34050, len:2202, log_data_len:297, seq:8720, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:31.593734] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=74] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18050, miss_cnt:0, cache_read_size:2208019, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:31.612086] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D8F1-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:31.612450] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D8F1-0-0] [lt=361] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=22031, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:31.612477] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D8F1-0-0] [lt=24] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=22, avg_log_submit_to_flush_time=6034, avg_log_submit_to_slide_time=22009) [2025-01-08 00:51:31.612812] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:31.645426] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690F1D-0-0] [lt=13] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.789499998092651367e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=894, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.777316481322638708e+01) [2025-01-08 00:51:31.645908] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3962, avg_group_log_size=396) [2025-01-08 00:51:31.647206] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.715344] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=29] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:31.715371] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:31.715470] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84989-0-0] [lt=14] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:31.715491] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84989-0-0] [lt=17] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:31.720611] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:31.720691] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44658, [delay/us]:prepare:7, schedule:229, submit:290, rt:1056, total:1583) [2025-01-08 00:51:31.720710] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:31.720771] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44658, [delay/us]: prepare:7, schedule:229, submit:290, device:1056, total:1583) [2025-01-08 00:51:31.720785] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:31.720828] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:73077, [delay/us]:prepare:10, schedule:203, submit:179, rt:1414, total:1808) [2025-01-08 00:51:31.720838] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:31.720873] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:73077, [delay/us]: prepare:10, schedule:203, submit:179, device:1414, total:1808) [2025-01-08 00:51:31.720884] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:31.720906] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16239, [delay/us]:prepare:7, schedule:67, submit:1, rt:900, total:976) [2025-01-08 00:51:31.720928] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:2, hold_mem:16384 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4381, iops:43, bw:186752, [delay/us]:prepare:8, schedule:278, submit:313, rt:2348, total:2949) [2025-01-08 00:51:31.720937] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=6, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49998, free_result_cnt=49998, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:31.721056] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=108] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16237, [delay/us]: prepare:7, schedule:67, submit:1, device:900, total:976) [2025-01-08 00:51:31.721117] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=49] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4382, avg_iops:43, avg_bw:186727, [delay/us]: prepare:8, schedule:278, submit:313, device:2348, total:2949) [2025-01-08 00:51:31.721129] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:31.721164] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4059, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:489, total:497) [2025-01-08 00:51:31.721244] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:31.721283] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4059, [delay/us]: prepare:0, schedule:0, submit:600000000, device:489, total:497) [2025-01-08 00:51:31.721307] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:31.721327] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:31.721336] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:31.721344] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:31.721321] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:31.721352] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=244kB/s, net_out=230kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:31.729454] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:31.747323] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.800125] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=27] ====== check clog disk timer task ====== [2025-01-08 00:51:31.800162] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=29] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:31.805338] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1003}, {id:1}, {id:1002}]) [2025-01-08 00:51:31.805633] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:31.805651] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=78] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:31.805724] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=37] gc stale ls task succ [2025-01-08 00:51:31.806773] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:51:31.814074] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:31.814152] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=74][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:31.816242] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE8BA-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326291815563000, start_id=1736326291815563000, end_id=1736326291815563001) [2025-01-08 00:51:31.816256] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] get number(ret=0, service_type_=0, range=1, base_id=1736326291815563000, start_id=1736326291815563001, end_id=1736326291815563002) [2025-01-08 00:51:31.817321] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46E7-0-0] [lt=23][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18245, log_task={header:{begin_lsn:{lsn:2220662}, end_lsn:{lsn:2220784}, log_id:18245, min_scn:{val:1736326291324638003, v:0}, max_scn:{val:1736326291324638003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2220540}, prev_proposal_id:1, committed_end_lsn:{lsn:2220662}, data_checksum:1715074492, accum_checksum:2992374992, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326291711016, freeze_ts:1736326291711016, submit_ts:1736326291711022, flushed_ts:1736326291714800, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:3778}, fs_cb_begin_ts=1736326291817296, log_life_time=106280) [2025-01-08 00:51:31.824047] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870F4-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:31.824099] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:31.824111] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:31.827920] INFO pn_ratelimit (group.c:653) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=109] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:31.829105] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3E0-0-0] [lt=41][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:31.831554] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D07A-0-0] [lt=267][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:31.835286] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:31.835317] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=19] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:31.835333] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=174) [2025-01-08 00:51:31.835367] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=23] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:31.835389] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=15] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=207) [2025-01-08 00:51:31.837279] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185436-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:31.839061] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=649809920, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=423931904, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:31.839136] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=40] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:31.839155] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=17] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2853, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:31.839340] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1116487680, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2104737792, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:31.839453] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=77] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:31.839472] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:31.845021] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:31.846637] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1609BCC3-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326291843794000, start_id=1736326291843794001, end_id=1736326291843794002) [2025-01-08 00:51:31.847262] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:31.847491] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:31.847994] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.848048] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43370785}], scn_array=[cnt:1, {val:1736326291831715000, v:0}], curr_size=589, accum_size=7891, time_cost=13373) [2025-01-08 00:51:31.850634] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:31.851294] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58540][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=55] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:51:31.853833] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46494][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1709C9ED-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326291853446000, start_id=1736326291853446000, end_id=1736326291853446001) [2025-01-08 00:51:31.856496] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:31.856714] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:31.857029] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=19] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=4159488) [2025-01-08 00:51:31.858839] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=147] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:31.858970] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=73][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84364562}], scn_array=[cnt:1, {val:1736326291843794000, v:0}], curr_size=586, accum_size=1415, time_cost=12221) [2025-01-08 00:51:31.868796] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B2-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:31.868843] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B2-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326291324638004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326291324638004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:31.894899] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=79] PNIO [ratelimit] time: 1736326291894895, bytes: 0, bw: 0.000000 MB/s, add_ts: 1003594, add_bytes: 0 [2025-01-08 00:51:31.901076] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.901076] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.901146] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.901264] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:31.907590] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50509][T1002_L0_G0][T1002][YB42C0A89F8A-00062B2D1BC9C7BE-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326291907248000, start_id=1736326291907248000, end_id=1736326291907248001) [2025-01-08 00:51:31.917958] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58554][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:31.918085] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:31.921101] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18699171}], scn_array=[cnt:1, {val:1736326291907248000, v:0}], curr_size=545, accum_size=1155, time_cost=11780) [2025-01-08 00:51:31.923739] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:31.925966] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB02B-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326291925384000, start_id=1736326291925384000, end_id=1736326291925384001) [2025-01-08 00:51:31.931095] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:31.931458] INFO [COMMON] wash (ob_kvcache_store.cpp:356) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=60] Wash time detail, (compute_wash_size_time=219, refresh_score_time=106, wash_time=0) [2025-01-08 00:51:31.931608] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=105] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:31.932477] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] get_min_unreplayed_log_info(lsn={lsn:2220906}, scn={val:1736326291324638005, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326291821772, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2220906}, next_to_submit_scn_:{val:1736326291324638005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2220784}, end_lsn:{lsn:2220906}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2221028}, readable_log_tail:{lsn:2221028}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64883740, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326291324638004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326291324638004, v:0}, data_checksum:3348837161, flag:54324}}, init_mode_version:0, accumulate_checksum:3529311704, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326291324638004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18154, miss_cnt:0, cache_read_size:2220906, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:31.935068] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=46] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:31.948382] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:31.956209] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=32] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:328K/74:300091 disk:328K/74:300091 t1m2:56K/12:204290 t1m2g0:56K/12:204290 t1002m2:196K/46:265535 t1002m2g0:196K/46:265535 t1001m2:76K/16:470465 t1001m2g0:76K/16:470465 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:31.978351] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CAA-0-0] [lt=61][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:31.980107] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CAA-0-0] [lt=94][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:32.013335] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=196, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:32.013781] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=443] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:32.015242] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BAB-0-0] [lt=53] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:32.020016] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=74, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:32.020065] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=2066, avg_group_log_size=206) [2025-01-08 00:51:32.020087] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2066) [2025-01-08 00:51:32.020103] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:32.020438] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=81, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:32.020462] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:32.020478] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1098) [2025-01-08 00:51:32.020490] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:32.025447] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A93AC-0-0] [lt=57] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:51:32.026054] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:32.034806] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=102] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:32.040554] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DC6-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9819) [2025-01-08 00:51:32.041965] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=2736, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:32.041987] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9819) [2025-01-08 00:51:32.042022] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=217, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:32.042124] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=10, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:32.042147] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=4859, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:32.042157] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=78, avg_log_submit_to_flush_time=3117, avg_log_submit_to_slide_time=4781) [2025-01-08 00:51:32.042241] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=21, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:32.042379] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9819, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:32.042680] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=43] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326291312444003, v:0}, new_max_version={val:1736326291312444003, v:0}, last_min_version={val:1736326291312444001, v:0}, last_max_version={val:1736326291312444001, v:0}, total_time_us=10099, delta_us=730234) [2025-01-08 00:51:32.042735] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326291312444003, v:0}, min_version={val:1736326291312444003, v:0}, max_version={val:1736326291312444003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:32.042772] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1736326291907248000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18699838}, palf_committed_end_scn_:{val:1736326291907248001, v:0}, last_check_scn_:{val:1736326291907248001, v:0}, max_applied_cb_scn_:{val:1736326291907248000, v:0}}) [2025-01-08 00:51:32.042814] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=38] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18699838}, readable_log_tail:{lsn:18699838}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48404930, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:32.042930] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=94] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:32.048065] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=23] LogLoopThread round_cost_time(us)(round_cost_time=16) [2025-01-08 00:51:32.048439] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.080952] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:32.081038] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=64] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=99) [2025-01-08 00:51:32.123896] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:32.124002] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8E-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326292239675, keepalive_ts:-1}) [2025-01-08 00:51:32.124104] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8E-0-0] [lt=41] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326292239675, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:32.126502] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC877A-0-0] [lt=43] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:32.126541] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC877A-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6401, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:32.131987] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:32.139351] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C79C28D-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9776, avg_group_log_size=651) [2025-01-08 00:51:32.148589] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.157835] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3040, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:32.158071] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=227] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=231, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:32.158126] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=33, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:32.214075] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58525][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46ED-0-0] [lt=37] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1210) [2025-01-08 00:51:32.215916] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C685-0-0] [lt=56][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193797129747, cache_obj->added_lc()=false, cache_obj->get_object_id()=13647, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:32.217540] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC877C-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:32.217566] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC877C-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=32557, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:32.217577] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC877C-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=11, avg_log_submit_to_flush_time=2503, avg_log_submit_to_slide_time=32545) [2025-01-08 00:51:32.217645] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1210, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:32.217677] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18157, miss_cnt:0, cache_read_size:2221272, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:32.218055] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=58, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:32.222842] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2595, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:32.222924] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:32.222946] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=93, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:32.223467] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=10, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:32.224356] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC877D-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:32.224389] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC877D-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=5138, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:32.224459] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:32.234834] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=64, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:32.234887] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10365) [2025-01-08 00:51:32.237532] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=45] ====== traversal_flush timer task ====== [2025-01-08 00:51:32.237733] INFO [MDS] flush (mds_table_mgr.cpp:195) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853D8-0-0] [lt=23] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326291907248002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326289309715001, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326300087570000, latest_log_ts:{val:1736326283797570001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.237931] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853D8-0-0] [lt=161] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326287305763004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326289309715001, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.238021] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853D8-0-0] [lt=62] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326287318696004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326289309715001, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.238104] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:32.238103] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853D8-0-0] [lt=56] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326287308713003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326289309715001, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.238158] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853D8-0-0] [lt=53] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:32.248992] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.278825] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:32.284258] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5271693056, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:32.284297] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:32.284310] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a1eb0) [2025-01-08 00:51:32.284319] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5271693056, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:32.308779] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3991-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:32.308907] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3991-0-0] [lt=56] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:32.308936] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3991-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:32.308983] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3991-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18147, log_task={header:{begin_lsn:{lsn:2208751}, end_lsn:{lsn:2208873}, log_id:18147, min_scn:{val:1736326292049655001, v:0}, max_scn:{val:1736326292049655001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2208629}, prev_proposal_id:1, committed_end_lsn:{lsn:2208751}, data_checksum:940009244, accum_checksum:2646436692, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326292208620, freeze_ts:1736326292208620, submit_ts:1736326292208634, flushed_ts:1736326292218024, gen_to_freeze cost time:0, gen_to_submit cost time:14, submit_to_flush cost time:9390}, fs_cb_begin_ts=1736326292308954, log_life_time=100334) [2025-01-08 00:51:32.310407] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2373, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:32.310494] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:32.310534] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=824, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:32.313677] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46EF-0-0] [lt=54] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:32.313723] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A46EF-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1210, avg_group_log_size=121) [2025-01-08 00:51:32.315055] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE8BD-0-0] [lt=41] handle gts request statistics(total_rt=2994, total_cnt=282, avg_rt=1.057950530035335746e+01, avg_cnt=2.819999999999999929e+01) [2025-01-08 00:51:32.315753] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2234, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:32.317250] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:32.319486] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.319541] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.325034] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=35] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1352608, replace_num=15728) [2025-01-08 00:51:32.325100] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=64] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:32.325690] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=56] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4003, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:32.325714] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:32.325733] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=554, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:32.325794] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:32.326377] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8781-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=102, avg_log_submit_to_flush_time=4669, avg_log_submit_to_slide_time=6619) [2025-01-08 00:51:32.326494] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1643, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:32.328131] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58554][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A93B6-0-0] [lt=41] init point thread id with(&point=0x5572d50e57c0, idx_=1210, point=[thread id=58554, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58554) [2025-01-08 00:51:32.332473] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:32.334237] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:32.334272] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-01-08 00:51:32.334284] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:32.334299] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:32.334328] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193796247942, flush_io_finished_ret_:0, flush_io_finished_round_:1955, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:32.347498] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:32.347649] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:32.349127] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.356906] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:32.356963] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:32.357489] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=101] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:32.359124] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:32.379443] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=41] current monitor number(seq_=-1) [2025-01-08 00:51:32.394072] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC2-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:32.394272] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC2-0-0] [lt=195] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:32.395741] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1736326292395738, bytes: 878132803, bw: 0.313744 MB/s, add_ts: 1003046, add_bytes: 329987 [2025-01-08 00:51:32.415499] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=136] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:32.415605] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=79] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=134, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:32.415733] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=73, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:32.422113] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=37] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:32.427166] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] ====== traversal_flush timer task ====== [2025-01-08 00:51:32.427439] INFO [MDS] flush (mds_table_mgr.cpp:195) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EEC-0-0] [lt=121] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326292339815000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326299284252000, latest_log_ts:{val:1736326282955407001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.427568] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EEC-0-0] [lt=123] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:32.448590] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=126] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3962) [2025-01-08 00:51:32.448680] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:32.449563] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.450067] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3962) [2025-01-08 00:51:32.450172] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3962) [2025-01-08 00:51:32.452006] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C66-0-0] [lt=21] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:32.452042] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C66-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6240, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:32.452056] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C66-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=824, avg_log_gen_to_submit_time=945, avg_log_submit_to_flush_time=3160, avg_log_submit_to_slide_time=5295) [2025-01-08 00:51:32.452151] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3962, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:32.457732] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=50] gc stale ls task succ [2025-01-08 00:51:32.459059] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.459570] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.461811] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=48] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:32.461876] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=59] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:32.461893] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=15] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:32.461905] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:32.464609] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=132, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}]) [2025-01-08 00:51:32.465019] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=402] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:32.519534] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.525346] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:32.533010] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:32.541855] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.544245] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.549707] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.555071] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB609-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:32.555115] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB609-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326292316531001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326292316531001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:32.556935] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=103, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:32.562036] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1A8BB609-0-0] [lt=41] need check schema(new_schema_version=1736326191428712, cached_tenant_schema_version=1736326191428712, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=select * from __all_ls_recovery_stat where ls_id = ? and tenant_id = ? for update) [2025-01-08 00:51:32.570728] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=24] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:32.603644] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=43] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8721, offset=2476175}, end_cursor:ObLogCursor{file_id=1, log_id=8722, offset=2476480}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b38050, len:1905, log_data_len:305, seq:8721, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:32.604183] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E60-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:32.604279] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E60-0-0] [lt=73] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326292316531003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326292316531003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:32.613927] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A93BD-0-0] [lt=28] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:32.614026] INFO [LIB] stat (utility.h:1210) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A93BD-0-0] [lt=98] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23098, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:32.614044] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58539][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A93BD-0-0] [lt=15] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=26, avg_log_submit_to_flush_time=3568, avg_log_submit_to_slide_time=23071) [2025-01-08 00:51:32.614418] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:32.614456] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18061, miss_cnt:0, cache_read_size:2209361, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:32.616234] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=47] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8722, offset=2476480}, end_cursor:ObLogCursor{file_id=1, log_id=8723, offset=2476777}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b3c050, len:1600, log_data_len:297, seq:8722, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:32.616792] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E60-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:32.617241] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E60-0-0] [lt=266] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326292316531003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326292316531003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:32.617753] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:32.621014] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E60-0-0] [lt=100] submit log success(service_type_=0, last_id=1736326309284252000, limited_id=1736326309284252000) [2025-01-08 00:51:32.621059] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E60-0-0] [lt=43] submit log with lock success(service_type_=0, last_id=1736326309284252000, limited_id=1736326309284252000) [2025-01-08 00:51:32.621698] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:32.621783] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=83] try gc retain ctx [2025-01-08 00:51:32.621793] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-01-08 00:51:32.621974] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=160] finish update min start scn(max_decided_scn={val:1736326292316531003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326277123056004, v:0}, keep_alive_scn_:{val:1736326287305763002, v:0}, update_ts_:1736326292621529}) [2025-01-08 00:51:32.623258] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326290311284001, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326300087570000, latest_log_ts:{val:1736326283797570001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:32.623498] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=81] finish update min start scn(max_decided_scn={val:1736326292316531003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326277123056004, v:0}, keep_alive_scn_:{val:1736326287305763002, v:0}, update_ts_:1736326292622742}) [2025-01-08 00:51:32.624525] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326287305763004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326290311284001, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:32.624674] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=68] finish update min start scn(max_decided_scn={val:1736326292324052001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326285326952004, v:0}, keep_alive_scn_:{val:1736326290323279004, v:0}, update_ts_:1736326292623961}) [2025-01-08 00:51:32.625865] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326287318696004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326289309715001, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:32.625934] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=62] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:32.625909] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=36] update id meta(ret=0, service_type=0, limited_id=1736326309284252000, latest_scn={val:1736326292617740001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309284252000, latest_log_ts:{val:1736326292617740001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:32.625960] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=50] handle submit callback(service_type_=0, success=true, used_ts=4947, log_ts={val:1736326292617740001, v:0}, limited_id=1736326309284252000) [2025-01-08 00:51:32.625972] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=12] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:32.625981] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:32.625952] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] get_min_unreplayed_log_info(lsn={lsn:2209361}, scn={val:1736326292316531003, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326292613904, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2209361}, next_to_submit_scn_:{val:1736326292316531003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2209239}, end_lsn:{lsn:2209361}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2209361}, readable_log_tail:{lsn:2209361}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64895407, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326292316531002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326292316531002, v:0}, data_checksum:2124447694, flag:22578}}, init_mode_version:0, accumulate_checksum:1857309981, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326292316531002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18061, miss_cnt:0, cache_read_size:2209361, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:32.626080] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=124] finish update min start scn(max_decided_scn={val:1736326292316531002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326284308664001, v:0}, keep_alive_scn_:{val:1736326289313850001, v:0}, update_ts_:1736326292625105}) [2025-01-08 00:51:32.626995] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326287308713003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326289309715001, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:32.631913] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326292631103000, start_id=1736326292631103000, end_id=1736326292631103001) [2025-01-08 00:51:32.634249] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58556][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:32.634315] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58556][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DD0-0-0] [lt=38] init point thread id with(&point=0x5572d50e5840, idx_=1212, point=[thread id=58556, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58556) [2025-01-08 00:51:32.634702] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:51:32.636345] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=43] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8723, offset=2476777}, end_cursor:ObLogCursor{file_id=1, log_id=8724, offset=2477074}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b40050, len:1303, log_data_len:297, seq:8723, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:32.637487] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8724, offset=2477074}, end_cursor:ObLogCursor{file_id=1, log_id=8725, offset=2477371}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b48050, len:1006, log_data_len:297, seq:8724, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:32.651458] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.651890] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690F2C-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.789499998092651367e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=894, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.777316481322638708e+01) [2025-01-08 00:51:32.652567] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3505, avg_group_log_size=350) [2025-01-08 00:51:32.652593] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] submit log success(service_type_=0, last_id=1736326309254996000, limited_id=1736326309254996000) [2025-01-08 00:51:32.652604] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] submit log with lock success(service_type_=0, last_id=1736326309254996000, limited_id=1736326309254996000) [2025-01-08 00:51:32.652840] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BAC-0-0] [lt=17][errcode=-4006] not init(ret=-4006) [2025-01-08 00:51:32.652901] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:480) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BAC-0-0] [lt=41] finish scan map holding items(scan_cnt=0) [2025-01-08 00:51:32.653076] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BAD-0-0] [lt=11] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:32.655753] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=6] update id meta(ret=0, service_type=0, limited_id=1736326309254996000, latest_scn={val:1736326292647912001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309254996000, latest_log_ts:{val:1736326292647912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:32.655906] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=152] handle submit callback(service_type_=0, success=true, used_ts=3313, log_ts={val:1736326292647912001, v:0}, limited_id=1736326309254996000) [2025-01-08 00:51:32.655923] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=16] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:32.655933] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:32.659093] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326292658410000, start_id=1736326292658410000, end_id=1736326292658410001) [2025-01-08 00:51:32.667739] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BECF-0-0] [lt=35] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:32.672323] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5990) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000) [2025-01-08 00:51:32.672358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.672432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.672499] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.672563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.672712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.672783] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.672846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.672908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.672970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673253] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673440] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673594] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.673837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674160] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674308] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=85] get multi version start(ls_id={id:1}, tablet_id={id:50355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674436] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101098}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674731] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674905] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101036}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.674964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675079] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101064}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675136] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675270] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675336] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101080}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675499] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675568] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675628] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675686] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675744] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675802] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675860] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.675961] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676026] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676085] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101083}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676143] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676262] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101062}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676325] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676560] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101018}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676676] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676735] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676850] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101034}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.676966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677024] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677140] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101084}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677274] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677339] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677449] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677513] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677572] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677689] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101068}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677747] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677805] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.677979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678160] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678299] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678363] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678421] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678479] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678537] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678653] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678768] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678826] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101073}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678883] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678940] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.678998] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679179] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101099}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679320] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=87] get multi version start(ls_id={id:1}, tablet_id={id:451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679466] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101092}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679644] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679875] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679933] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.679990] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680048] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101031}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680106] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680299] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680481] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680774] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680897] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.680960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681247] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681370] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101067}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681486] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681602] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681659] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681774] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101027}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.681969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.682027] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.682085] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.682143] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.682944] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:60465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683269] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683332] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683391] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683450] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683566] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101082}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683624] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.683913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684092] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684305] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684543] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684604] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684664] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684725] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684784] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684844] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.684964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685023] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101057}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685141] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685200] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685333] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685395] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685454] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685513] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685572] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685690] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685748] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685807] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101033}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.685941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686003] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686122] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686182] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686323] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=85] get multi version start(ls_id={id:1}, tablet_id={id:60181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686384] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101066}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686444] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686503] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686621] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686680] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686914] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.686973] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687032] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687092] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101030}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101094}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687279] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101020}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687407] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101044}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687466] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687525] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687584] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687761] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687937] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.687997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688055] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688114] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688174] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688307] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688368] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688427] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688486] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101013}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688605] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688664] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688782] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.688960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101072}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689135] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:60432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689267] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101088}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:101069}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689492] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689547] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=10] get multi version start(ls_id={id:1}, tablet_id={id:50357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689691] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689790] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689889] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.689985] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690042] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690129] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690280] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690413] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690518] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:50249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101060}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690718] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690782] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101100}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.690992] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691146] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691273] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691368] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101022}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691430] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691515] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691576] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691759] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.691962] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.692049] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.692105] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.692164] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.692294] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=72] get multi version start(ls_id={id:1}, tablet_id={id:60412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.692356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.692415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.692474] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.692532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.692587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693282] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693489] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101035}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693550] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:5}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693636] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.693947] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101058}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694132] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:101081}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101086}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694513] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101052}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694790] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:101019}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.694972] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695071] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101079}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695312] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=85] get multi version start(ls_id={id:1}, tablet_id={id:50272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695572] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:50333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695691] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695749] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695807] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:1}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695924] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.695982] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696099] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696270] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696511] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696572] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696761] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.696918] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697112] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697293] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:60419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697474] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=52] get multi version start(ls_id={id:1}, tablet_id={id:60148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697646] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697896] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.697955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698257] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698323] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698442] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698558] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698731] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698905] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.698964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699022] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699139] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699339] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699411] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699529] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699705] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699764] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.699999] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700174] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700298] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700363] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700539] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700829] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.700945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701003] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101043}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701225] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701340] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701451] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701519] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701579] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701637] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701694] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701808] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701923] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.701980] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702037] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702094] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702302] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702399] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:101032}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702533] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101089}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702756] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.702951] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=77] get multi version start(ls_id={id:1}, tablet_id={id:50356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703172] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703384] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703446] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703505] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703564] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703623] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703740] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703836] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.703957] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704192] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704411] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704761] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101028}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704935] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.704994] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705052] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705110] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705167] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705297] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=73] get multi version start(ls_id={id:1}, tablet_id={id:411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101021}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705474] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705533] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705591] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705707] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705765] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705823] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705940] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.705998] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706174] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706287] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=60] get multi version start(ls_id={id:1}, tablet_id={id:209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706522] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706648] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706764] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101037}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706880] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706938] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.706997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707055] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707114] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101054}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707172] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707315] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707434] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707629] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707695] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707783] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.707843] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101041}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=178] get multi version start(ls_id={id:1}, tablet_id={id:525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708248] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708325] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708390] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101097}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708451] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708512] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708573] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101024}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.708754] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709100] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709162] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709302] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709368] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709429] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709489] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101055}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.709966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:60458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710366] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710427] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710488] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101063}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.710974] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.711355] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:3}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.711433] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.711697] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=152] get multi version start(ls_id={id:1}, tablet_id={id:60112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.711767] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.711829] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.711889] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.711949] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.712009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101040}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.712131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.712439] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.712569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.712701] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.712822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.712888] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.713012] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:101047}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.713074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.713135] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.713351] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.713693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.713881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.713960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714146] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714275] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714568] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=108] get multi version start(ls_id={id:1}, tablet_id={id:101059}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714695] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=71] get multi version start(ls_id={id:1}, tablet_id={id:50131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714757] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.714933] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715001] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715122] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715182] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715582] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715827] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.715950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.716019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.716146] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.716404] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=154] get multi version start(ls_id={id:1}, tablet_id={id:498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.716502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.716723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.716848] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.717050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=143] get multi version start(ls_id={id:1}, tablet_id={id:50345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.717113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.717173] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.717319] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.720303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.720429] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.720504] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.720723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.720960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=177] get multi version start(ls_id={id:1}, tablet_id={id:60343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721231] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721366] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721590] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721652] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101046}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.721952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=343] get multi version start(ls_id={id:1}, tablet_id={id:60105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722483] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722604] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722665] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722725] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722845] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.722966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723085] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723146] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101045}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723465] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723525] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723585] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723705] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723765] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.723978] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724124] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724297] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101078}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724619] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724734] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:32.724740] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724780] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48951, [delay/us]:prepare:42, schedule:119, submit:64, rt:2157, total:2383) [2025-01-08 00:51:32.724811] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724795] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:32.724872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101049}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724896] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=84] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48950, [delay/us]: prepare:42, schedule:119, submit:64, device:2157, total:2383) [2025-01-08 00:51:32.724911] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:32.724932] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724976] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:73423, [delay/us]:prepare:7, schedule:475, submit:201, rt:1945, total:2629) [2025-01-08 00:51:32.724993] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.724988] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:32.725026] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:73423, [delay/us]: prepare:7, schedule:475, submit:201, device:1945, total:2629) [2025-01-08 00:51:32.725037] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:32.725059] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16316, [delay/us]:prepare:6, schedule:40, submit:1, rt:1840, total:1887) [2025-01-08 00:51:32.725054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.725083] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:2, hold_mem:16384 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4203, iops:38, bw:159083, [delay/us]:prepare:6, schedule:224, submit:265, rt:1922, total:2418) [2025-01-08 00:51:32.725091] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=7, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49998, free_result_cnt=49997, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:32.725119] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16317, [delay/us]: prepare:6, schedule:40, submit:1, device:1840, total:1887) [2025-01-08 00:51:32.725135] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4204, avg_iops:38, avg_bw:159104, [delay/us]: prepare:6, schedule:224, submit:265, device:1922, total:2418) [2025-01-08 00:51:32.725145] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:32.725167] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4079, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1356, total:1366) [2025-01-08 00:51:32.725285] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:32.725334] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4079, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1356, total:1366) [2025-01-08 00:51:32.725355] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:32.725373] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:32.725382] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:32.725390] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:32.725397] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=333kB/s, net_out=235kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:32.725456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.725548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:50209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.725613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101101}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.725675] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.725760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.725809] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:32.725820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.725978] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726040] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726100] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726382] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726786] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101091}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.726982] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.727042] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.727347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.727568] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:101039}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.727643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:101112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.727707] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101029}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.727768] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.728017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.728222] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.728297] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.728654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.729270] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.729341] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.729402] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.729460] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.729519] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.730012] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2221760}], scn_array=[cnt:1, {val:1736326292324052003, v:0}], curr_size=122, accum_size=366, time_cost=10283) [2025-01-08 00:51:32.729578] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.732480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=2838] get multi version start(ls_id={id:1}, tablet_id={id:433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.732570] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.732633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.732728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.732790] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.732888] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.732986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.733530] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:32.736949] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=3518] get multi version start(ls_id={id:1}, tablet_id={id:50413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.737057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.737128] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.737188] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.737402] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=152] get multi version start(ls_id={id:1}, tablet_id={id:100004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.737504] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.737795] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.737948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=92] get multi version start(ls_id={id:1}, tablet_id={id:60500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:60389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738120] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738312] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738423] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:101095}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738488] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101070}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738875] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:101075}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.738963] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739121] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739253] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739351] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101077}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739412] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739505] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739692] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101071}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.739856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101093}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.740165] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:101012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.740369] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=124] get multi version start(ls_id={id:1}, tablet_id={id:420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.740455] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=30] get multi version start(ls_id={id:1}, tablet_id={id:50220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.740582] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.740801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.740946] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:101065}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741294] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:50495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741431] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741689] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741802] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.741985] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742044] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742104] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742321] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101014}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742539] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742662] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=110] get multi version start(ls_id={id:1}, tablet_id={id:50443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742910] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.742970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743091] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743150] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743210] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743388] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743460] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:60456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743824] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:60473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743889] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.743950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.744009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.744083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.744372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.744555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.744622] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.744681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.744759] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.744822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.745089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.745178] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.745414] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=181] get multi version start(ls_id={id:1}, tablet_id={id:60283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.745502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.745599] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.745693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.745788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746035] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746260] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746346] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101050}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746577] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746634] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746692] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746749] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746807] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.746978] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.747035] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.747092] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101056}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.747150] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.747258] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.747569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=256] get multi version start(ls_id={id:1}, tablet_id={id:60511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.749762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.749882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.749946] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.750045] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.750134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.750283] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.750380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.750484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:60207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.750549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:101023}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.750635] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.750788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.750929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:101007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.751399] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101048}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.751512] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=57] get multi version start(ls_id={id:1}, tablet_id={id:330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.751728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=160] get multi version start(ls_id={id:1}, tablet_id={id:101026}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.751784] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.751870] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.751965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752179] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752320] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=85] get multi version start(ls_id={id:1}, tablet_id={id:50354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752411] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752466] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=54] get multi version start(ls_id={id:1}, tablet_id={id:513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752923] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.752987] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753152] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753414] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:50106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753515] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101025}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753632] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:60421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753925] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.753986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754046] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:4}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754107] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754167] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754291] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754354] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754414] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754835] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.754954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755013] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755073] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755132] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755275] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755406] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101085}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755467] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755779] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755880] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.755980] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756178] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756283] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.756306] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756490] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101087}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756525] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=44] ====== tenant freeze timer task ====== [2025-01-08 00:51:32.756559] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756680] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756862] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756922] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.756978] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757130] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757190] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757326] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:50501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757388] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757485] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101051}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:100006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.757975] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758039] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758267] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758419] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101061}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.758842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101076}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.759115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.759256] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.759324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.759386] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.759445] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.760012] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.760094] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.760160] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.760341] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.760463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.760535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.760664] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.760801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.760903] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761105] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761203] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761425] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101074}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761871] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.761963] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:101002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762024] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101053}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762271] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762436] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762522] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762578] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762731] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762786] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60322}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.762932] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763283] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763436] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763496] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763554] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763731] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763790] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50322}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763848] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.763966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.764025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.764085] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.764144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.764202] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.764491] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=218] get multi version start(ls_id={id:1}, tablet_id={id:505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.764581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.764719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.764822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765164] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=52] get multi version start(ls_id={id:1}, tablet_id={id:112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=132] get multi version start(ls_id={id:1}, tablet_id={id:60241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:60359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765553] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.765906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766006] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766109] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766173] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766360] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=127] get multi version start(ls_id={id:1}, tablet_id={id:60169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766462] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:322}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766521] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766610] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101042}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766836] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.766958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767046] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767109] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767171] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=230] get multi version start(ls_id={id:1}, tablet_id={id:60492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767539] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767656] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767747] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101096}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767811] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.767997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768278] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768689] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768790] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768889] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.768980] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.769097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=61] get multi version start(ls_id={id:1}, tablet_id={id:50148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.769155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.769288] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:60222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.769388] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.769487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101038}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.769588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.769773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770026] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:50486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770280] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:60238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770427] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101090}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770607] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=108] get multi version start(ls_id={id:1}, tablet_id={id:373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770827] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324484334180000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324484334180000, max_merged_snapshot=1) [2025-01-08 00:51:32.770874] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:116) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EED-0-0] [lt=41] SSTableGCTask(cost_ts=98890) [2025-01-08 00:51:32.776691] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=32] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:32.778454] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=27] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:32.778541] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=18] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326284334180000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:32.805078] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=41] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:32.805842] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:32.806733] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=54] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:32.806860] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=49] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326285633619000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:32.806929] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=67] gc stale ls task succ [2025-01-08 00:51:32.806970] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=56] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1002}, {id:1}, {id:1001}]) [2025-01-08 00:51:32.807124] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326282796685, end_time=1736326292807123, dag_status={}) [2025-01-08 00:51:32.807150] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:32.807162] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:32.807169] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:32.807176] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:32.807268] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=91] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:32.807278] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:32.807284] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:32.807291] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:32.807298] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:32.807305] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807316] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807332] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807340] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807348] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807356] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807364] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807372] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807380] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807388] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807396] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807404] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807412] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807420] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807428] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807436] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807443] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807451] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807459] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807467] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807475] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807483] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807490] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807498] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807506] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807514] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807522] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807530] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807538] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807546] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807554] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807561] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807569] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807577] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807584] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807592] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807600] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807608] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807616] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807624] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807632] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807640] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807648] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807656] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807664] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807671] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807679] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807687] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807694] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807702] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807710] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807718] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807725] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807733] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807799] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=65] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807808] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807816] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807823] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807831] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807894] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807903] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807911] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807919] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807927] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807934] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807942] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807950] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807958] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807966] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807974] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807982] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807990] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:32.807999] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:32.808006] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:32.808013] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:32.808018] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:32.808023] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:32.808029] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:32.808034] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:32.808039] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:32.808044] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:32.808049] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:32.808055] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:32.808093] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:32.808125] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=18] gc stale ls task succ [2025-01-08 00:51:32.809746] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:32.809770] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:32.812528] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3E3-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:32.817594] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84647-0-0] [lt=53] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2025-01-08 00:51:32.818976] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84647-0-0] [lt=25] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2025-01-08 00:51:32.818995] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1010) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84647-0-0] [lt=17] no transfer partition job [2025-01-08 00:51:32.819008] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1040) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84647-0-0] [lt=7] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer=true) [2025-01-08 00:51:32.819902] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB870FA-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:32.820094] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:32.820110] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:32.820445] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8C2-0-0] [lt=112] get number(ret=0, service_type_=0, range=1, base_id=1736326292817796000, start_id=1736326292817796000, end_id=1736326292817796001) [2025-01-08 00:51:32.821604] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:284) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84647-0-0] [lt=10] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-01-08 00:51:32.821640] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:493) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84647-0-0] [lt=24] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2025-01-08 00:51:32.821668] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:157) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84647-0-0] [lt=20] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=3, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1003}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone3", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1736324440080711, last_statistic_bg_stat_time=1736326192603087, last_statistic_schema_version=1736326192037600, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-01-08 00:51:32.823613] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D07D-0-0] [lt=100][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:32.825132] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2209483}], scn_array=[cnt:1, {val:1736326292316531004, v:0}], curr_size=122, accum_size=610, time_cost=14982) [2025-01-08 00:51:32.837472] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:32.837512] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=29] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=88) [2025-01-08 00:51:32.839432] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185437-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:32.839609] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=38] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:32.839632] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:51:32.839642] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:32.839655] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:32.839668] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193796751443, flush_io_finished_ret_:0, flush_io_finished_round_:1951, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:32.839715] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:32.839730] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-01-08 00:51:32.839736] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:32.839744] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:32.839754] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193796751455, flush_io_finished_ret_:0, flush_io_finished_round_:3779, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:32.839942] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] success to update min reserved snapshot(reserved_snapshot=1736324485742394000, duration=1800, snapshot_gc_ts=1736326285742394000) [2025-01-08 00:51:32.845297] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=33] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:32.846440] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=54] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:32.848510] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43380734}], scn_array=[cnt:1, {val:1736326292832989000, v:0}], curr_size=588, accum_size=8020, time_cost=11796) [2025-01-08 00:51:32.848656] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=16] update reserved snapshot finished(cost_ts=8717, reserved_snapshot=1736324485742394000) [2025-01-08 00:51:32.848894] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:32.849528] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:32.850129] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853DF-0-0] [lt=25] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:32.850156] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853DF-0-0] [lt=19] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:32.854790] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:32.855955] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=24] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:32.856041] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=18] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326285742394000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:32.856467] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.858801] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:32.858899] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:32.859828] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:32.859940] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=64] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:32.861463] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857B5-0-0] [lt=25][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193797775296, cache_obj->added_lc()=false, cache_obj->get_object_id()=13648, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:32.862497] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857B6-0-0] [lt=44][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:32.866260] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857B8-0-0] [lt=24][errcode=-4201] hashtable not init [2025-01-08 00:51:32.871540] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=72] ====== traversal_flush timer task ====== [2025-01-08 00:51:32.871688] INFO [MDS] flush (mds_table_mgr.cpp:195) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856F6-0-0] [lt=28] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326292848760000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309254996000, latest_log_ts:{val:1736326292647912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.871788] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D173856F6-0-0] [lt=93] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:32.890588] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=47] ====== checkpoint timer task ====== [2025-01-08 00:51:32.891406] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:32.891448] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=41] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:32.891566] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326290311284001, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326300087570000, latest_log_ts:{val:1736326283797570001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.891644] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=74] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:32.891663] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:32.891677] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326283797570001, v:0}) [2025-01-08 00:51:32.891685] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:32.891699] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:32.891717] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:32.891729] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:32.891748] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:32.891768] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:32.892519] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:32.892551] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=31] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:32.892589] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326287305763004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326290311284001, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.892647] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=56] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:32.892659] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:32.892671] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:32.892685] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:32.892700] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2188246}, scn={val:1736326292820856000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:32.893539] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=796) [2025-01-08 00:51:32.894636] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=26] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8725, offset=2477371}, end_cursor:ObLogCursor{file_id=1, log_id=8726, offset=2477668}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b44050, len:709, log_data_len:297, seq:8725, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:32.894683] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326292820856000, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:32.894707] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:32.894726] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:32.895526] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:32.895564] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=38] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:32.895607] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=20] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326287318696004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326290311284001, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.895669] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=60] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:32.895683] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:32.895697] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:32.895711] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:32.895724] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2222004}, scn={val:1736326292324052004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:32.896839] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.896875] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.896952] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8726, offset=2477668}, end_cursor:ObLogCursor{file_id=1, log_id=8727, offset=2477965}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b4c050, len:412, log_data_len:297, seq:8726, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:32.896989] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326292324052004, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:32.897011] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:32.897030] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:32.897230] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.897926] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:32.897960] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=34] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:32.898002] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326287308713003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326290311284001, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:32.898116] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=112] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:32.898130] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:32.898145] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:32.898264] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=105] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:32.898285] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2209483}, scn={val:1736326292316531003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:32.898406] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:32.900308] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B3-0-0] [lt=29] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:32.900337] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B3-0-0] [lt=18] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326292324052004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326292324052004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:32.902538] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=101] PNIO [ratelimit] time: 1736326292902535, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007640, add_bytes: 0 [2025-01-08 00:51:32.911719] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58538][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC399D-0-0] [lt=32][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18153, log_task={header:{begin_lsn:{lsn:2209483}, end_lsn:{lsn:2209605}, log_id:18153, min_scn:{val:1736326292316531004, v:0}, max_scn:{val:1736326292316531004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2209361}, prev_proposal_id:1, committed_end_lsn:{lsn:2209483}, data_checksum:940009244, accum_checksum:2448095760, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326292810038, freeze_ts:1736326292810038, submit_ts:1736326292810051, flushed_ts:1736326292825295, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:15244}, fs_cb_begin_ts=1736326292911669, log_life_time=101631) [2025-01-08 00:51:32.912571] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=24] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8727, offset=2477965}, end_cursor:ObLogCursor{file_id=1, log_id=8728, offset=2478262}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b50050, len:4211, log_data_len:297, seq:8727, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:32.912871] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326292316531003, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:32.912986] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:32.913013] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:32.913025] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:32.913771] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] BlockGCTimerTask success(ret=0, cost_time_us=24, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:32.918533] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C2-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326292936051, keepalive_ts:-1}) [2025-01-08 00:51:32.918564] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C2-0-0] [lt=29] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326292936051, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:32.919005] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C1-0-0] [lt=15] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326292935969, keepalive_ts:-1}) [2025-01-08 00:51:32.919031] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58539][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C1-0-0] [lt=26] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326292935969, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:32.924132] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1001, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:17978, max_lsn:2188246, max_scn:{val:1736326292820856000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2188246}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2188124}, last_submit_end_lsn:{lsn:2188246}, last_submit_log_id:17978, last_submit_log_pid:1, max_flushed_lsn:{lsn:2188124}, max_flushed_end_lsn:{lsn:2188246}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2188246}, last_slide_log_id:17978, last_slide_scn:{val:1736326292820856000, v:0}, last_slide_lsn:{lsn:2188124}, last_slide_end_lsn:{lsn:2188246}, last_slide_log_pid:1, last_slide_log_accum_checksum:1978386811, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924f6a530}, [StateMgr]={this:0x2ad924f6e830, self:"192.168.159.137:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:17979, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324447115984, palf_role_change_cb:0x2ad924f752f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924f6bc70}, [ModeMgr]={palf_id:1001, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324447762546, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436756804001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/meta", dir_fd:177, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924f70a38, flashback_version:0}, log_storage:{log_tail:{lsn:2188246}, readable_log_tail:{lsn:2188246}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64916522, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1736324447114985, this:0x2ad924f6ef70}, [Reconfirm]={palf_id:1001, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:1"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324447117136, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924f6e9b0}) [2025-01-08 00:51:32.924448] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=308] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326292826364, last_flushed_end_lsn:{lsn:2188246}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326292830259, last_flushed_end_lsn:{lsn:2188246}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326292831932, last_flushed_end_lsn:{lsn:2188246}}]) [2025-01-08 00:51:32.924474] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1001, self="192.168.159.137:2882", log_id=17979, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326292826364, last_flushed_end_lsn:{lsn:2188246}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326292830259, last_flushed_end_lsn:{lsn:2188246}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326292831932, last_flushed_end_lsn:{lsn:2188246}}]) [2025-01-08 00:51:32.925452] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:32.925972] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:32.927261] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2F-0-0] [lt=47] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326292944632, keepalive_ts:-1}) [2025-01-08 00:51:32.927312] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF2F-0-0] [lt=49] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326292944632, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:32.929230] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58539][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:32.931601] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE1-0-0] [lt=23] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:32.931676] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE1-0-0] [lt=66] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=78) [2025-01-08 00:51:32.934104] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=84] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:32.935995] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1006624, clean_num=31457) [2025-01-08 00:51:32.956576] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:32.990486] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CAB-0-0] [lt=40][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:32.992001] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CAB-0-0] [lt=98][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:32.994504] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:801) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D1718512A-0-0] [lt=27] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, location:{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, location:{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}]) [2025-01-08 00:51:33.009157] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=27] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:320K/76:215441 disk:320K/76:215441 t1m2:40K/10:87832 t1m2g0:40K/10:87832 t1002m2:208K/50:243025 t1002m2g0:208K/50:243025 t1001m2:72K/16:208996 t1001m2g0:72K/16:208996 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:33.014058] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=76, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:33.014222] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=162] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=4, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:33.024313] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=72] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326289000207, last_first_register_time_us=1736324449183670, register_time_us=1736326289000207, curr_time_us=1736326293024266) [2025-01-08 00:51:33.024418] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=55] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326293024370, last_submit_register_req_time_us=1736326293024370, register_parent_reason=1) [2025-01-08 00:51:33.024436] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:33.024800] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=71, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:33.024833] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:33.024852] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:33.024867] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:33.024918] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:33.024934] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:33.024945] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:33.024955] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:33.025794] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:33.026096] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:33.030050] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F8-0-0] [lt=11] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326293049080, keepalive_ts:-1}) [2025-01-08 00:51:33.030083] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F8-0-0] [lt=33] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326293049080, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:33.038291] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:33.042274] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DD6-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10537) [2025-01-08 00:51:33.042384] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DD6-0-0] [lt=43] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=10, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:33.042400] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DD6-0-0] [lt=15] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4857, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:33.043585] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326292316531003, v:0}, new_max_version={val:1736326292316531003, v:0}, last_min_version={val:1736326292316531002, v:0}, last_max_version={val:1736326292316531002, v:0}, total_time_us=7249, delta_us=727051) [2025-01-08 00:51:33.043620] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326292316531003, v:0}, min_version={val:1736326292316531003, v:0}, max_version={val:1736326292316531003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:33.044287] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58557][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:33.044794] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] runTimer to refresh locality_info(ret=0) [2025-01-08 00:51:33.046992] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:346) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=25] set_server_locality_array success(server_locality_array=[{inited:true, addr:"192.168.159.137:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322580423131, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.138:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581270010, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.139:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581175973, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-01-08 00:51:33.047064] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=59] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-01-08 00:51:33.047078] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=13] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-01-08 00:51:33.047092] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1001, primary_zone=RANDOM) [2025-01-08 00:51:33.047099] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] add_locality_zone success(locality_zone={tenant_id:1001, region_priority:18446744073709551615}) [2025-01-08 00:51:33.047112] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1002, primary_zone=RANDOM) [2025-01-08 00:51:33.047250] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=137] add_locality_zone success(locality_zone={tenant_id:1002, region_priority:18446744073709551615}) [2025-01-08 00:51:33.047269] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=12] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1001, region_priority:18446744073709551615}, {tenant_id:1002, region_priority:18446744073709551615}]}) [2025-01-08 00:51:33.047469] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852AD-0-0] [lt=16] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326288372474, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:33.047527] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852AD-0-0] [lt=55] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326288374043, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:33.047550] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852AD-0-0] [lt=23] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, renew_time:1736326288375643, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, renew_time:1736326288375644, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, renew_time:1736326288375646, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326288375639, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:33.052315] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread created(tenant_->id()=1001, group_id_=1, token=8) [2025-01-08 00:51:33.056932] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.118822] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF5D-0-0] [lt=38] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326293237243, keepalive_ts:-1}) [2025-01-08 00:51:33.118867] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF5D-0-0] [lt=43] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326293237243, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:33.126159] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:33.130288] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8798-0-0] [lt=101] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=39, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:33.130528] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8798-0-0] [lt=238] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5229, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:33.136538] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:33.142142] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1736326292940919000, v:0}, base_version={val:1736326292739398000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326292940919000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326293142124, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326292940919000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326293248027, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326292940919000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326293147509, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:33.144443] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=122] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9991, avg_group_log_size=624) [2025-01-08 00:51:33.145569] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT WRITE LOG COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=2432, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:33.145596] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10537) [2025-01-08 00:51:33.145615] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=237, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:33.145665] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=23, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:33.146958] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DD7-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=359, avg_log_gen_to_submit_time=446, avg_log_submit_to_flush_time=2711, avg_log_submit_to_slide_time=4277) [2025-01-08 00:51:33.147171] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10537, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:33.157245] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.159101] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1688, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:33.159157] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=536, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:33.159190] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=86] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:33.191974] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-01-08 00:51:33.192000] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:33.217202] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D911-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:33.225988] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1518, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:33.226017] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:33.226034] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=121, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:33.226134] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:33.228925] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC879C-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:33.228962] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC879C-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4143, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:33.229026] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:33.234242] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46100][TimerWK2][T0][YB42C0A89F89-00062B2D17E85446-0-0] [lt=57] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:33.238922] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=70, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:33.238957] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9402) [2025-01-08 00:51:33.241529] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C687-0-0] [lt=22][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193798155363, cache_obj->added_lc()=false, cache_obj->get_object_id()=13649, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:33.257398] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.257929] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A7C-0-0] [lt=35] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326293257892}) [2025-01-08 00:51:33.258000] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598D-0-0] [lt=20] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:33.260066] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598D-0-0] [lt=39] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326293260044, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326293260044, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326293260044, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:33.260164] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598D-0-0] [lt=97] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326293257892}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326293260044, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326293260044, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326293260044, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:33.271149] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=32][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:33.271476] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=114][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:33.271536] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=18][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:33.281635] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:33.283902] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] ====== check clog disk timer task ====== [2025-01-08 00:51:33.283931] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:33.284174] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=41] start to print blacklist info [2025-01-08 00:51:33.284764] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=96] ls blacklist refresh finish(cost_time=28587) [2025-01-08 00:51:33.284992] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5273692286, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:33.285092] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=29] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:33.285107] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5273692286, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:33.309379] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39A5-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:33.309457] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39A5-0-0] [lt=64] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:33.309478] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39A5-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:33.310740] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2690, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:33.310839] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:33.310889] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=233, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:33.310975] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=58, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:33.313818] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4703-0-0] [lt=30] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:33.313880] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4703-0-0] [lt=53] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:33.315983] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8C5-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326293314596000, start_id=1736326293314596000, end_id=1736326293314596001) [2025-01-08 00:51:33.317544] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D913-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:33.317580] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D913-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29876, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:33.317593] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D913-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=2662, avg_log_submit_to_slide_time=29862) [2025-01-08 00:51:33.317654] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:33.317683] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18168, miss_cnt:0, cache_read_size:2222614, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:33.319261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.319619] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.322462] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=26] ts response task statistics(alloc_count=60410, free_count=30204) [2025-01-08 00:51:33.322550] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=79] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:33.326791] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=112] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:33.328324] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1652, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:33.328376] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:33.328393] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=122, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:33.328440] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:33.330528] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58552][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC879E-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=85, avg_log_submit_to_flush_time=1841, avg_log_submit_to_slide_time=5079) [2025-01-08 00:51:33.331004] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:33.335361] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:33.337165] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:33.348577] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:33.351029] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D46-0-0] [lt=36] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:33.353434] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=17] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:33.353578] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:33.357530] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.358933] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:33.359085] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:33.360033] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=95] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:33.360181] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:33.395697] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC3-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:33.395738] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC3-0-0] [lt=37] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:33.399451] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=50] PNIO [ratelimit] time: 1736326293399448, bytes: 878373713, bw: 0.228900 MB/s, add_ts: 1003710, add_bytes: 240910 [2025-01-08 00:51:33.415088] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2259, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:33.439815] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=109] ====== tenant freeze timer task ====== [2025-01-08 00:51:33.451210] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] ====== tenant freeze timer task ====== [2025-01-08 00:51:33.451300] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=40] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71124160, tenant_id=1001) [2025-01-08 00:51:33.451330] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=20] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:33.454297] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3536) [2025-01-08 00:51:33.454341] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:33.455614] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3535) [2025-01-08 00:51:33.455698] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=94] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3536) [2025-01-08 00:51:33.458610] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:51:33.459326] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.459612] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.459868] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=50][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:33.460078] WDIAG read_multi_row (ob_table_access_helper.h:238) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:33.462490] INFO delete_row (ob_table_access_helper.h:339) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=46] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1001, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326293460125) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326293460125) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:33.462663] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.463439] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C70-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:33.463475] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C70-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5570, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:33.463488] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58527][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C70-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=232, avg_log_gen_to_submit_time=318, avg_log_submit_to_flush_time=2247, avg_log_submit_to_slide_time=5252) [2025-01-08 00:51:33.463627] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3535, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:33.513265] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] clear dtl interm result cost(us)(clear_cost=7260, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:33.515952] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:33.516055] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=209, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:33.516129] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=50, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:33.519283] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=43] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.529044] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=77] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:33.537765] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:33.541085] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.544429] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.559041] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=78, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:33.562809] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.578634] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60A-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:33.578755] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60A-0-0] [lt=51] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326293314596000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326293314596000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:33.591719] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB60A-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1736326293590779000, start_id=1736326293590779000, end_id=1736326293590779001) [2025-01-08 00:51:33.594508] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=25] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:33.643831] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:194) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] get wrs ts(ls_id={id:1}, delta=99824, log_service_ts=1736326293542950, min_tx_service_ts=4611686018427387, timestamp={val:1736326293542950000, v:0}) [2025-01-08 00:51:33.643889] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=51] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=748, Other_Error_Cnt=0, Submit_Succ_Cnt=211, last_scn="{val:1736326289514488001, v:0}", last_lsn={lsn:43348210}, last_gts={val:1736326293542950000, v:0}, min_start_scn="{val:1736326289419518000, v:0}", min_start_status=2, sys_ls_end_scn={val:0, v:0}) [2025-01-08 00:51:33.644010] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8728, offset=2478262}, end_cursor:ObLogCursor{file_id=1, log_id=8729, offset=2478567}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b54050, len:3914, log_data_len:305, seq:8728, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:33.644387] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E61-0-0] [lt=93] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:33.644441] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E61-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326293321615002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326293321615002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:33.655514] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] no need to update min/max version(new_version={val:1736326293251811000, v:0}, min_version={val:1736326293251811000, v:0}, max_version={val:1736326293251811000, v:0}) [2025-01-08 00:51:33.659316] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=178] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3529, avg_group_log_size=352) [2025-01-08 00:51:33.663033] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=81] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:33.663119] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.664197] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8729, offset=2478567}, end_cursor:ObLogCursor{file_id=1, log_id=8730, offset=2478864}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b58050, len:3609, log_data_len:297, seq:8729, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:33.664582] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E61-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:33.664688] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E61-0-0] [lt=46] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326293321615003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326293321615003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:33.672800] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8730, offset=2478864}, end_cursor:ObLogCursor{file_id=1, log_id=8731, offset=2479161}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b64050, len:3312, log_data_len:297, seq:8730, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:33.673639] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8731, offset=2479161}, end_cursor:ObLogCursor{file_id=1, log_id=8732, offset=2479458}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b6c050, len:3015, log_data_len:297, seq:8731, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:33.673877] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18071, miss_cnt:0, cache_read_size:2210581, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:33.716923] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D91A-0-0] [lt=81] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=22, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:33.716949] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D91A-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=37594, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:33.716959] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D91A-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=2874, avg_log_submit_to_slide_time=37575) [2025-01-08 00:51:33.717323] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:33.717568] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:33.717609] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=40] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:33.717664] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498A-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:33.717698] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498A-0-0] [lt=31] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:33.728571] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:33.728671] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=65] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, bw:40801, [delay/us]:prepare:14, schedule:34, submit:74, rt:1254, total:1377) [2025-01-08 00:51:33.728689] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:33.728743] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40802, [delay/us]: prepare:14, schedule:34, submit:74, device:1254, total:1377) [2025-01-08 00:51:33.728756] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:33.728851] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4973, iops:14, bw:69367, [delay/us]:prepare:33, schedule:144, submit:824, rt:1589, total:2591) [2025-01-08 00:51:33.728866] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:33.728907] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4974, avg_iops:14, avg_bw:69362, [delay/us]: prepare:33, schedule:144, submit:824, device:1589, total:2591) [2025-01-08 00:51:33.728919] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:33.728943] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:7, bw:32641, [delay/us]:prepare:5, schedule:245, submit:1, rt:1425, total:1676) [2025-01-08 00:51:33.728966] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:163206, [delay/us]:prepare:6, schedule:89, submit:107, rt:1674, total:1877) [2025-01-08 00:51:33.728975] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12240, [delay/us]:prepare:2, schedule:1758, submit:86, rt:348, total:2195) [2025-01-08 00:51:33.728983] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:33.729011] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4681, avg_iops:7, avg_bw:32640, [delay/us]: prepare:5, schedule:245, submit:1, device:1425, total:1676) [2025-01-08 00:51:33.729026] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12240, [delay/us]: prepare:2, schedule:1758, submit:86, device:348, total:2195) [2025-01-08 00:51:33.729034] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:163203, [delay/us]: prepare:6, schedule:89, submit:107, device:1674, total:1877) [2025-01-08 00:51:33.729043] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:33.729062] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4080, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:329, total:336) [2025-01-08 00:51:33.729130] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:33.729188] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=51] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4080, [delay/us]: prepare:0, schedule:0, submit:600000000, device:329, total:336) [2025-01-08 00:51:33.729273] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:33.729684] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=111] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:33.729976] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:33.730070] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=90] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:33.730092] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:33.730112] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SERVER] net_in=241kB/s, net_out=230kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:33.734435] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=33] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:33.738702] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1736326293314596003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3, epoch_tstamp:1736326293738624}, version_delta=424105) [2025-01-08 00:51:33.739268] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=152] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:33.741387] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] BlockGCTimerTask success(ret=0, cost_time_us=17, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:33.741497] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085163-0-0] [lt=25] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:33.748043] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43390008}, readable_log_tail:{lsn:43390008}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23714760, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:33.757722] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690F3B-0-0] [lt=25] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.789499998092651367e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=894, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.777316481322638708e+01) [2025-01-08 00:51:33.758116] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] get number(ret=0, service_type_=0, range=1, base_id=1736326293757707000, start_id=1736326293757707000, end_id=1736326293757707001) [2025-01-08 00:51:33.767469] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.768002] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=42] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:33.769361] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84371474}], scn_array=[cnt:1, {val:1736326293757707000, v:0}], curr_size=585, accum_size=829, time_cost=10506) [2025-01-08 00:51:33.801471] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=29] ====== check clog disk timer task ====== [2025-01-08 00:51:33.801507] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=31] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:33.806308] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:33.808056] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:202) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=7] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2025-01-08 00:51:33.808079] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=19] finish to resize log external storage handler(current_ts=1736326293808075, tenant_max_cpu=4, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:33.808092] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=12] snapshot_traffic_info once [2025-01-08 00:51:33.808325] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=59] gc stale ls task succ [2025-01-08 00:51:33.810453] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:33.810497] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:33.810676] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1002}, {id:1001}]) [2025-01-08 00:51:33.810780] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:33.810822] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=9] gc stale ls task succ [2025-01-08 00:51:33.814476] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A470D-0-0] [lt=84][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18264, log_task={header:{begin_lsn:{lsn:2222980}, end_lsn:{lsn:2223102}, log_id:18264, min_scn:{val:1736326293324054003, v:0}, max_scn:{val:1736326293324054003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2222858}, prev_proposal_id:1, committed_end_lsn:{lsn:2222980}, data_checksum:1715074492, accum_checksum:3638633740, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326293714270, freeze_ts:1736326293714270, submit_ts:1736326293714277, flushed_ts:1736326293718402, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:4125}, fs_cb_begin_ts=1736326293814468, log_life_time=100198) [2025-01-08 00:51:33.815608] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D080-0-0] [lt=30][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:33.818274] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3E6-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:33.820676] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87100-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:33.820720] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:33.820732] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:33.821743] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8C8-0-0] [lt=221] submit log success(service_type_=0, last_id=1736326310087570000, limited_id=1736326310087570000) [2025-01-08 00:51:33.821809] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8C8-0-0] [lt=64] submit log with lock success(service_type_=0, last_id=1736326310087570000, limited_id=1736326310087570000) [2025-01-08 00:51:33.823649] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=56] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326293823608], gts:1736326293822919000, latest_srr:[mts=1736326293823608]}) [2025-01-08 00:51:33.823691] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326293823679], gts:1736326293822919000, latest_srr:[mts=1736326293823679]}) [2025-01-08 00:51:33.823909] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [46205][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13] is logging [2025-01-08 00:51:33.823969] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [46205][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=58] submit log with lock(ret=-4023, service_type_=0, last_id=1736326310087570000, limited_id=1736326310087570000) [2025-01-08 00:51:33.824034] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326293823904], gts:1736326293822919000, latest_srr:[mts=1736326293823904]}) [2025-01-08 00:51:33.824626] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=30] update id meta(ret=0, service_type=0, limited_id=1736326310087570000, latest_scn={val:1736326293821063001, v:0}, *this={tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326291315024002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326310087570000, latest_log_ts:{val:1736326293821063001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:33.824684] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=57] handle submit callback(service_type_=0, success=true, used_ts=2942, log_ts={val:1736326293821063001, v:0}, limited_id=1736326310087570000) [2025-01-08 00:51:33.824695] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:33.824703] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:33.829522] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:263) [50687][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D2008458E-0-0] [lt=20] ObArchiveSender is running(thread_index=4) [2025-01-08 00:51:33.841833] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185438-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:33.842244] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:33.842294] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=41] temporary file module use no blocks [2025-01-08 00:51:33.842308] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:33.842397] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=86] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:33.842415] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193797753539, flush_io_finished_ret_:0, flush_io_finished_round_:1952, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:33.842508] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:33.842527] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-01-08 00:51:33.842533] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:33.842541] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:33.842551] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193797753616, flush_io_finished_ret_:0, flush_io_finished_round_:3780, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:33.846012] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=34] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:33.853878] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=32] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=12478464) [2025-01-08 00:51:33.853968] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=43] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:33.859030] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:33.859434] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:33.860161] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=41] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:33.860554] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=43] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:33.867551] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.896520] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.896955] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.897139] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.897311] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:33.908715] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=82] PNIO [ratelimit] time: 1736326293908711, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006176, add_bytes: 0 [2025-01-08 00:51:33.931566] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:33.934146] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=27] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:33.934873] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:33.935276] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58525][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=38] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:33.939787] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:33.946684] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B4-0-0] [lt=52] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:33.946736] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B4-0-0] [lt=35] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326293825501000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326293825501000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:33.949926] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=24] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:33.961697] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=91] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=17271387, this={dio_aligned_buf:{buf_write_offset:3237, buf_padding_size:859, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:70906, truncate_used_ts:17461}, log_block_size:67108864, io_fd:{first_id:-1, second_id:152, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=17268736, buf_len=586, total_write_size=39753, total_write_size_after_dio=442368, ob_pwrite_used_ts=274086, count=99, each_pwrite_cost=2768) [2025-01-08 00:51:33.968340] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:33.998721] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C689-0-0] [lt=39][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193798912552, cache_obj->added_lc()=false, cache_obj->get_object_id()=13650, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:34.001886] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CAC-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:34.005453] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CAC-0-0] [lt=97][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:34.013143] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=38] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:276K/66:77358 disk:276K/66:77358 t1m2:36K/8:23425 t1m2g0:36K/8:23425 t1002m2:176K/44:77263 t1002m2g0:176K/44:77263 t1001m2:64K/14:108475 t1001m2g0:64K/14:108475 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:34.016836] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=252] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=52, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:34.016999] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=161] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:34.021107] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=26] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:34.026039] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:19626, max_lsn:18702153, max_scn:{val:1736326293822919000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:18702153}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:18702031}, last_submit_end_lsn:{lsn:18702153}, last_submit_log_id:19626, last_submit_log_pid:1, max_flushed_lsn:{lsn:18702031}, max_flushed_end_lsn:{lsn:18702153}, max_flushed_log_pid:1, committed_end_lsn:{lsn:18702153}, last_slide_log_id:19626, last_slide_scn:{val:1736326293822919000, v:0}, last_slide_lsn:{lsn:18702031}, last_slide_end_lsn:{lsn:18702153}, last_slide_log_pid:1, last_slide_log_accum_checksum:1086179096, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8fc454530}, [StateMgr]={this:0x2ad8fc458830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:19627, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324399554770, palf_role_change_cb:0x2ad8fc45f2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8fc455c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324399758968, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/meta", dir_fd:175, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8fc45aa38, flashback_version:0}, log_storage:{log_tail:{lsn:18702153}, readable_log_tail:{lsn:18702153}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48402615, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324399554279, this:0x2ad8fc458f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324399555093, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8fc4589b0}) [2025-01-08 00:51:34.026390] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=343] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326293937039, last_flushed_end_lsn:{lsn:18702153}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326293937289, last_flushed_end_lsn:{lsn:18702153}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326293936322, last_flushed_end_lsn:{lsn:18702153}}]) [2025-01-08 00:51:34.026416] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=19627, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326293937039, last_flushed_end_lsn:{lsn:18702153}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326293937289, last_flushed_end_lsn:{lsn:18702153}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326293936322, last_flushed_end_lsn:{lsn:18702153}}]) [2025-01-08 00:51:34.026461] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1002, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18267, max_lsn:2223468, max_scn:{val:1736326293825501001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2223346}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2223346}, last_submit_end_lsn:{lsn:2223468}, last_submit_log_id:18267, last_submit_log_pid:1, max_flushed_lsn:{lsn:2223224}, max_flushed_end_lsn:{lsn:2223346}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2223346}, last_slide_log_id:18266, last_slide_scn:{val:1736326293825501000, v:0}, last_slide_lsn:{lsn:2223224}, last_slide_end_lsn:{lsn:2223346}, last_slide_log_pid:1, last_slide_log_accum_checksum:186151777, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad92435e530}, [StateMgr]={this:0x2ad924362830, self:"192.168.159.137:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.138:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18267, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad9243692f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326293024370, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326293024370, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad92435fc70}, [ModeMgr]={palf_id:1002, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436803442001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/meta", dir_fd:184, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924364a38, flashback_version:0}, log_storage:{log_tail:{lsn:2223346}, readable_log_tail:{lsn:2223346}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64881422, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:-1, this:0x2ad924362f70}, [Reconfirm]={palf_id:1002, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9243629b0}) [2025-01-08 00:51:34.026658] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=193][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:34.026672] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:34.026681] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:34.026691] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] current log_task status(palf_id=1002, self="192.168.159.137:2882", log_id=18267, log_task={header:{begin_lsn:{lsn:2223346}, end_lsn:{lsn:2223468}, log_id:18267, min_scn:{val:1736326293825501001, v:0}, max_scn:{val:1736326293825501001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2223224}, prev_proposal_id:1, committed_end_lsn:{lsn:2223346}, data_checksum:1715074492, accum_checksum:1542145319, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1736326294023831, freeze_ts:1736326294023831, submit_ts:1736326294023868, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:37, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:51:34.026720] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326290007935, last_first_register_time_us=1736324454612308, register_time_us=1736326290007935, curr_time_us=1736326294026719) [2025-01-08 00:51:34.026772] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326294026734, last_submit_register_req_time_us=1736326294026734, register_parent_reason=1) [2025-01-08 00:51:34.026786] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:34.034024] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=89, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:34.034131] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=96] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1226, avg_group_log_size=122) [2025-01-08 00:51:34.034272] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=136] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1226) [2025-01-08 00:51:34.034289] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:34.034362] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=42, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:34.034437] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:34.034472] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:34.034485] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:34.036355] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A93E8-0-0] [lt=104] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1226) [2025-01-08 00:51:34.040134] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A93E9-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:34.043387] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:34.044508] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=102] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:34.044979] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2189588}], scn_array=[cnt:1, {val:1736326293822919002, v:0}], curr_size=122, accum_size=976, time_cost=10553) [2025-01-08 00:51:34.046445] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:34.046972] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58523][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:51:34.049844] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18702153}], scn_array=[cnt:1, {val:1736326293822919001, v:0}], curr_size=122, accum_size=860, time_cost=15606) [2025-01-08 00:51:34.050280] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58526][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A7C-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=8686) [2025-01-08 00:51:34.050440] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46495][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C68E-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326294050157000, start_id=1736326294050157000, end_id=1736326294050157001) [2025-01-08 00:51:34.051255] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=18, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:34.051296] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4746, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:34.052029] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=110] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326293314596004, v:0}, new_max_version={val:1736326293314596004, v:0}, last_min_version={val:1736326293314596002, v:0}, last_max_version={val:1736326293314596002, v:0}, total_time_us=10758, delta_us=737431) [2025-01-08 00:51:34.052064] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326293314596004, v:0}, min_version={val:1736326293314596004, v:0}, max_version={val:1736326293314596004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:34.054843] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=74] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:34.057899] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58559][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:34.058058] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:51:34.068529] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.126611] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF5E-0-0] [lt=49] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326294242886, keepalive_ts:-1}) [2025-01-08 00:51:34.126665] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF5E-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326294242886, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:34.132135] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:34.136397] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC7-0-0] [lt=109] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326294255168, keepalive_ts:-1}) [2025-01-08 00:51:34.136450] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC7-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326294255168, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:34.137440] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A93EB-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:34.137467] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A93EB-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6031, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:34.140472] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=102] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:34.151826] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=2743, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:34.151888] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=59] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9275) [2025-01-08 00:51:34.151919] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=817, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:34.151982] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=22, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:34.155879] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=38] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:34.156508] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DE5-0-0] [lt=18] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=113, avg_log_submit_to_flush_time=3601, avg_log_submit_to_slide_time=5094) [2025-01-08 00:51:34.156618] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=89] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9275, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:34.156824] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] no need to update min/max version(new_version={val:1736326293757707000, v:0}, min_version={val:1736326293757707000, v:0}, max_version={val:1736326293757707000, v:0}) [2025-01-08 00:51:34.165469] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=147] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=3097, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:34.165529] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=253, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:34.166065] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:34.168677] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.222559] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4715-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:34.222618] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4715-0-0] [lt=34][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18268, log_task={header:{begin_lsn:{lsn:2223468}, end_lsn:{lsn:2223590}, log_id:18268, min_scn:{val:1736326293825501002, v:0}, max_scn:{val:1736326293825501002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2223346}, prev_proposal_id:1, committed_end_lsn:{lsn:2223468}, data_checksum:1715074492, accum_checksum:4250081605, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326294122364, freeze_ts:1736326294122364, submit_ts:1736326294122375, flushed_ts:1736326294126061, gen_to_freeze cost time:0, gen_to_submit cost time:11, submit_to_flush cost time:3686}, fs_cb_begin_ts=1736326294222597, log_life_time=100233) [2025-01-08 00:51:34.236853] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2899, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:34.236956] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=101] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:34.236980] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=371, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:34.237009] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=125] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:34.237579] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87B9-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:34.237604] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87B9-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5099, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:34.237684] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:34.243699] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get number(ret=0, service_type_=0, range=1, base_id=1736326294243638000, start_id=1736326294243638000, end_id=1736326294243638001) [2025-01-08 00:51:34.243937] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=94, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:34.243976] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9864, avg_group_log_size=657) [2025-01-08 00:51:34.243996] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9275) [2025-01-08 00:51:34.262261] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46507][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C68F-0-0] [lt=31][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193799176093, cache_obj->added_lc()=false, cache_obj->get_object_id()=13651, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:34.268799] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=82] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:34.268835] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.281811] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:34.285856] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5275574766, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:34.285978] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=116][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:34.285995] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8df7cfe60) [2025-01-08 00:51:34.286004] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5275574766, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:34.309890] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39B9-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:34.309946] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39B9-0-0] [lt=44] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:34.309965] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39B9-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:34.312355] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2099, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:34.312386] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:34.312409] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1011, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:34.312476] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=76, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:34.319293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.319776] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.322431] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4717-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:34.322487] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58552][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4717-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:34.324530] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:34.325128] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=139] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=508, get_gts_with_stc_cnt=3, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=3, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:34.325266] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D927-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:34.325289] INFO [LIB] stat (utility.h:1210) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D927-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=32049, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:34.325299] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58552][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D927-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=2458, avg_log_submit_to_slide_time=32035) [2025-01-08 00:51:34.325373] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=57] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:34.325399] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18178, miss_cnt:0, cache_read_size:2223834, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:34.332425] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=108] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:34.334549] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:34.334715] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=158] temporary file module use no blocks [2025-01-08 00:51:34.334729] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:34.334743] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:34.334782] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=37] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193798248256, flush_io_finished_ret_:0, flush_io_finished_round_:1957, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:34.336778] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=245] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3113, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:34.336820] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1226) [2025-01-08 00:51:34.336853] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=424, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:34.338014] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:34.338396] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87BD-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=14, avg_log_gen_to_submit_time=143, avg_log_submit_to_flush_time=3810, avg_log_submit_to_slide_time=5600) [2025-01-08 00:51:34.338502] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1226, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:34.339843] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=46] blacklist_loop exec finished(cost_time=64, is_enabled=true, send_cnt=0) [2025-01-08 00:51:34.340841] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:34.354224] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=43] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:34.354421] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:34.359453] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=66] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:34.359682] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=38] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:34.360734] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=50] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:34.360987] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=101] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:34.362398] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58559][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6910-0-0] [lt=27] init point thread id with(&point=0x5572d50e5940, idx_=1216, point=[thread id=58559, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58559) [2025-01-08 00:51:34.368904] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.379814] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=47] current monitor number(seq_=-1) [2025-01-08 00:51:34.380246] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:34.380284] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=32] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:34.380298] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:34.380309] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:34.383139] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:34.384054] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=911] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:34.398300] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC4-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:34.398390] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC4-0-0] [lt=86] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:34.407134] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1736326294407131, bytes: 878651466, bw: 0.262866 MB/s, add_ts: 1007683, add_bytes: 277753 [2025-01-08 00:51:34.423406] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2115, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:34.426851] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] LogLoopThread round_cost_time(us)(round_cost_time=25) [2025-01-08 00:51:34.442350] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326293821063004, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=621277, in_cluster_service=true, cluster_version={val:1736326293821063001, v:0}, min_cluster_version={val:1736326293821063001, v:0}, max_cluster_version={val:1736326293821063001, v:0}, get_cluster_version_err=0, cluster_version_delta=621277, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18478, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=522359867, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:34.445503] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=29] LogLoopThread round_cost_time(us)(round_cost_time=14) [2025-01-08 00:51:34.458741] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=40] gc stale ls task succ [2025-01-08 00:51:34.459469] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.459561] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.461657] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=91] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3395) [2025-01-08 00:51:34.461701] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:34.462883] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2932) [2025-01-08 00:51:34.463061] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3395) [2025-01-08 00:51:34.464322] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6911-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:34.464349] INFO [LIB] stat (utility.h:1210) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6911-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=5438, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:34.464360] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58527][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6911-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=33, avg_log_gen_to_submit_time=188, avg_log_submit_to_flush_time=3377, avg_log_submit_to_slide_time=5249) [2025-01-08 00:51:34.464504] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2932, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:34.469073] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.517014] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=39] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:34.519533] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=237] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.523838] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:34.523945] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=81] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=275, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:34.524087] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=72, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:34.533402] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=48] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1368336, replace_num=15728) [2025-01-08 00:51:34.533432] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:34.538259] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58560][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:34.538369] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58560][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87C3-0-0] [lt=78] init point thread id with(&point=0x5572d50e59c0, idx_=1218, point=[thread id=58560, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58560) [2025-01-08 00:51:34.540294] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:34.541340] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:34.542229] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.544054] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.546460] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:112) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AD86-0-0] [lt=110] adaptive update(Tenant ID=1001, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=392, Holding Memory(MB)=102, Tenant Remain Memory(MB)=482, Usable Remain Memory(MB)=289, Last Update Limit Timestamp=1736326294544299, Is Updated=true) [2025-01-08 00:51:34.549360] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=108] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:34.549425] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] get_max_applied_scn(scn={val:1736326294366039001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43396799}, palf_committed_end_scn_:{val:1736326294544299000, v:0}, last_check_scn_:{val:1736326294544299000, v:0}, max_applied_cb_scn_:{val:1736326294366039001, v:0}}) [2025-01-08 00:51:34.559392] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=118] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=110, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:34.561652] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:34.561677] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] try gc retain ctx [2025-01-08 00:51:34.561685] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-01-08 00:51:34.561705] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] get_max_applied_scn(scn={val:1736326294358680000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84374162}, palf_committed_end_scn_:{val:1736326294358680001, v:0}, last_check_scn_:{val:1736326294559167000, v:0}, max_applied_cb_scn_:{val:1736326294358680000, v:0}}) [2025-01-08 00:51:34.561780] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=70] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84374748}, readable_log_tail:{lsn:84374748}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49834788, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:34.562010] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=226] finish update min start scn(max_decided_scn={val:1736326294358680000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326284359372001, v:0}, keep_alive_scn_:{val:1736326289427266001, v:0}, update_ts_:1736326294561495}) [2025-01-08 00:51:34.563086] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309254996000, latest_log_ts:{val:1736326292647912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:34.569266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=69][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.570444] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:34.570523] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=77] try gc retain ctx [2025-01-08 00:51:34.570532] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-01-08 00:51:34.570616] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=63] finish update min start scn(max_decided_scn={val:1736326294544299000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326289419518000, v:0}, keep_alive_scn_:{val:1736326289514488001, v:0}, update_ts_:1736326294570063}) [2025-01-08 00:51:34.571507] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=97] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309284252000, latest_log_ts:{val:1736326292617740001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:34.601794] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60B-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:34.601838] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60B-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326294317339001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326294317339001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:34.627551] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=77] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:34.669530] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.676376] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8732, offset=2479458}, end_cursor:ObLogCursor{file_id=1, log_id=8733, offset=2479763}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b60050, len:2718, log_data_len:305, seq:8732, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:34.676771] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E62-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:34.676813] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E62-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326294324153003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326294324153003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:34.686754] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=92] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8733, offset=2479763}, end_cursor:ObLogCursor{file_id=1, log_id=8734, offset=2480060}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b5c050, len:2413, log_data_len:297, seq:8733, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:34.687343] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E62-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:34.687380] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E62-0-0] [lt=23] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326294324153003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326294324153003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:34.694436] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=99] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8734, offset=2480060}, end_cursor:ObLogCursor{file_id=1, log_id=8735, offset=2480357}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b70050, len:2116, log_data_len:297, seq:8734, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:34.695741] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=48] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8735, offset=2480357}, end_cursor:ObLogCursor{file_id=1, log_id=8736, offset=2480654}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b68050, len:1819, log_data_len:297, seq:8735, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:34.695960] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18081, miss_cnt:0, cache_read_size:2211792, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:34.697082] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=78] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:34.722020] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE39-0-0] [lt=15] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:34.722060] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE39-0-0] [lt=33] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:34.722091] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE39-0-0] [lt=28] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:34.724949] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=39] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:34.733625] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:34.735249] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:34.735326] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44758, [delay/us]:prepare:8, schedule:287, submit:71, rt:2168, total:2536) [2025-01-08 00:51:34.735343] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:34.735513] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=154] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44753, [delay/us]: prepare:8, schedule:287, submit:71, device:2168, total:2536) [2025-01-08 00:51:34.735530] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:34.735571] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:17, bw:77300, [delay/us]:prepare:7, schedule:234, submit:244, rt:1470, total:1956) [2025-01-08 00:51:34.735581] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:34.735618] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:77305, [delay/us]: prepare:7, schedule:234, submit:244, device:1470, total:1956) [2025-01-08 00:51:34.735636] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:34.735657] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16274, [delay/us]:prepare:4, schedule:14, submit:0, rt:1517, total:1536) [2025-01-08 00:51:34.735678] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4403, iops:40, bw:174953, [delay/us]:prepare:8, schedule:72, submit:402, rt:1818, total:2301) [2025-01-08 00:51:34.735686] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:34.735721] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16274, [delay/us]: prepare:4, schedule:14, submit:0, device:1517, total:1536) [2025-01-08 00:51:34.735737] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4403, avg_iops:40, avg_bw:174955, [delay/us]: prepare:8, schedule:72, submit:402, device:1818, total:2301) [2025-01-08 00:51:34.735747] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:34.735767] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4068, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:662, total:671) [2025-01-08 00:51:34.735791] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:34.735818] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4069, [delay/us]: prepare:0, schedule:0, submit:600000000, device:662, total:671) [2025-01-08 00:51:34.735836] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:34.735851] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:34.735859] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:34.735867] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:34.735875] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=258kB/s, net_out=222kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:34.741797] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=86] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:34.759013] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690F47-0-0] [lt=74] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.789499998092651367e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=894, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.777316481322638708e+01) [2025-01-08 00:51:34.759399] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4103, avg_group_log_size=373) [2025-01-08 00:51:34.760582] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=49] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:34.762810] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1736326294762474000, start_id=1736326294762474000, end_id=1736326294762474001) [2025-01-08 00:51:34.763161] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=19] ====== tenant freeze timer task ====== [2025-01-08 00:51:34.769731] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.773898] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:34.774084] WDIAG read_multi_row (ob_table_access_helper.h:238) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=103][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:34.775830] INFO delete_row (ob_table_access_helper.h:339) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=46] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326294774131) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326294774131) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:34.804945] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3E9-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:34.806486] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:34.807858] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D083-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:34.808685] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=13] gc stale ls task succ [2025-01-08 00:51:34.810971] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:34.811072] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:34.811168] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:34.811302] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=130][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:34.812113] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58561][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:34.812173] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39C3-0-0] [lt=26] init point thread id with(&point=0x5572d50e5a00, idx_=1219, point=[thread id=58561, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58561) [2025-01-08 00:51:34.812483] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1}, {id:1001}, {id:1003}]) [2025-01-08 00:51:34.813513] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:34.816027] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87106-0-0] [lt=137][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:34.816090] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:34.816103] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:34.824629] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] ====== checkpoint timer task ====== [2025-01-08 00:51:34.825804] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:34.825848] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=44] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:34.825955] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309284252000, latest_log_ts:{val:1736326292617740001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:34.826037] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=77] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:34.826067] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:34.826081] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326292617740001, v:0}) [2025-01-08 00:51:34.826088] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:34.826096] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:34.826108] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:34.826125] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:34.826138] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:34.826157] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:34.826177] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:34.826246] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:34.828369] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=65] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:34.840061] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=54] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=647712768, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=426029056, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:34.840299] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=199] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:34.840326] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=25] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2854, drift_size_=0, pre_drift_size=-6291456, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:34.840392] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1116487680, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2104737792, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:34.840447] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:34.840461] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:34.844057] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185439-0-0] [lt=103][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:34.844430] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=74] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:34.844516] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=42] temporary file module use no blocks [2025-01-08 00:51:34.844528] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:34.844549] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:34.844578] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193798756336, flush_io_finished_ret_:0, flush_io_finished_round_:1953, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:34.844622] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:34.844639] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-01-08 00:51:34.844645] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:34.844654] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:34.844664] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193798756413, flush_io_finished_ret_:0, flush_io_finished_round_:3781, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:34.845992] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AE13-0-0] [lt=65] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=107) [2025-01-08 00:51:34.846291] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326294642297000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=203986, in_cluster_service=true, cluster_version={val:1736326294544299000, v:0}, min_cluster_version={val:1736326294544299000, v:0}, max_cluster_version={val:1736326294544299000, v:0}, get_cluster_version_err=0, cluster_version_delta=301984, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18916, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=640483066, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:34.846641] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=14] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:34.855811] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:34.855962] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=65] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:34.859680] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:34.859892] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=70] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:34.860930] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:34.861425] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:34.869837] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=87][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.871065] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:34.872667] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326294872042000, start_id=1736326294872042000, end_id=1736326294872042001) [2025-01-08 00:51:34.876929] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:219) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:34.876975] INFO [COMMON] handle (memory_dump.cpp:538) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=43] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:34.878117] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] update_check_range(min_check_version=375, max_check_version=376, global_version=377) [2025-01-08 00:51:34.895678] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:34.897867] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326294887400000, start_id=1736326294887400000, end_id=1736326294887400001) [2025-01-08 00:51:34.898823] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.898861] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.899199] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=43] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.899391] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:34.907479] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=29] ====== checkpoint timer task ====== [2025-01-08 00:51:34.907601] INFO handle (memory_dump.cpp:665) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=34] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 152 262 0 1 2 68 1 0 1 4 1 1 0 1 5 43 5 0 1 7 1 2 0 1 8 48 1 0 1 12 7 3 0 1 13 1 1 0 1 14 1 2 0 1 15 1 1 0 1 18 4 3 0 1 20 1 2 0 500 0 94 237 0 500 7 7 4 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 2 1 0 500 17 9 7 0 500 22 79 62 0 500 23 23 10 0 508 0 2 10 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 99 230 0 1001 2 49 1 0 1001 4 1 1 0 1001 5 22 5 0 1001 8 30 1 0 1001 12 3 2 0 1001 13 1 1 0 1001 14 1 1 0 1001 16 1 1 0 1001 18 4 3 0 1001 20 2 2 0 1002 0 162 245 0 1002 2 59 1 0 1002 4 2 1 0 1002 5 49 6 0 1002 7 11 2 0 1002 8 47 1 0 1002 12 2 3 0 1002 13 1 1 0 1002 14 1 1 0 1002 18 9 3 0 1002 20 1 2 0 cost_time: 29517 [2025-01-08 00:51:34.907645] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant: 1, limit: 2,147,483,648 hold: 1,042,788,352 cache_hold: 213,909,504 cache_used: 213,909,504 cache_item_count: 102 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 447,782,912 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 142,606,336 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 95,862,784 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 100,663,296 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 14,680,064 limit= 429,496,720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:34.907998] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 447,782,912 used= 343,733,744 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,211,520 used= 31,196,160 count= 15 avg_used= 2,079,744 block_cnt= 15 chunk_cnt= 15 mod=CACHE_MAP_NODE [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 26,599,424 used= 26,451,520 count= 24 avg_used= 1,102,146 block_cnt= 24 chunk_cnt= 18 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,196,912 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 37 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,268,096 used= 11,228,160 count= 4 avg_used= 2,807,040 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,618,560 used= 7,541,760 count= 10 avg_used= 754,176 block_cnt= 10 chunk_cnt= 8 mod=ResourceGroup [MEMORY] hold= 7,077,888 used= 6,078,464 count= 122 avg_used= 49,823 block_cnt= 122 chunk_cnt= 24 mod=[T]char [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMemCU [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,416,064 used= 2,993,504 count= 139 avg_used= 21,536 block_cnt= 139 chunk_cnt= 28 mod=DiagnosticInfo [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,780,352 used= 2,663,280 count= 586 avg_used= 4,544 block_cnt= 570 chunk_cnt= 29 mod=WorkerMap [MEMORY] hold= 2,768,896 used= 2,720,000 count= 13 avg_used= 209,230 block_cnt= 13 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,640 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 272 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,394,368 used= 1,372,072 count= 75 avg_used= 18,294 block_cnt= 75 chunk_cnt= 11 mod=TabletMap [MEMORY] hold= 1,374,720 used= 1,308,288 count= 36 avg_used= 36,341 block_cnt= 36 chunk_cnt= 14 mod=CommSysVarFac [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 1,179,648 used= 1,175,808 count= 8 avg_used= 146,976 block_cnt= 8 chunk_cnt= 7 mod=SQLSessionInfo [MEMORY] hold= 960,192 used= 936,448 count= 102 avg_used= 9,180 block_cnt= 102 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 901,120 used= 442,640 count= 55 avg_used= 8,048 block_cnt= 55 chunk_cnt= 18 mod=[T]ObPerfEventR [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 502,016 used= 481,664 count= 106 avg_used= 4,544 block_cnt= 106 chunk_cnt= 26 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 262,144 used= 261,120 count= 4 avg_used= 65,280 block_cnt= 4 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 262,144 used= 150,272 count= 16 avg_used= 9,392 block_cnt= 16 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 203,840 used= 194,432 count= 49 avg_used= 3,968 block_cnt= 49 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 13 mod=LinearHashMapNo [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 12 mod=SqlSession [MEMORY] hold= 155,024 used= 2,177 count= 1,920 avg_used= 1 block_cnt= 19 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 154,432 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 13 mod=QueryAllocator [MEMORY] hold= 147,456 used= 131,072 count= 2 avg_used= 65,536 block_cnt= 2 chunk_cnt= 2 mod=MinorMergeMgr [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 105,664 used= 103,168 count= 13 avg_used= 7,936 block_cnt= 13 chunk_cnt= 9 mod=SqlSessiVarMap [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 7 mod=PlanVaIdx [MEMORY] hold= 92,192 used= 23,952 count= 339 avg_used= 70 block_cnt= 76 chunk_cnt= 26 mod=Coro [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 5 mod=HashNodeLCSta [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 54,720 used= 51,840 count= 15 avg_used= 3,456 block_cnt= 12 chunk_cnt= 11 mod=CACHE_INST [MEMORY] hold= 51,008 used= 39,648 count= 59 avg_used= 672 block_cnt= 58 chunk_cnt= 19 mod=[T]ObWarningBuf [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 39,120 used= 27,328 count= 61 avg_used= 448 block_cnt= 32 chunk_cnt= 11 mod=OMT_Worker [MEMORY] hold= 37,504 used= 3,984 count= 166 avg_used= 24 block_cnt= 67 chunk_cnt= 27 mod=[T]MemoryContex [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PartCtx [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,640 used= 15,872 count= 4 avg_used= 3,968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnMemNode [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,640 used= 6,320 count= 12 avg_used= 526 block_cnt= 12 chunk_cnt= 8 mod=SqlSessiQuerSql [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=SqlSessionSbloc [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 7,520 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 6 mod=PackStateMap [MEMORY] hold= 7,520 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 6 mod=SequenceMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 8 mod=ContextsMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 5 mod=SequenceIdMap [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,848 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 5 mod=DICollector [MEMORY] hold= 4,864 used= 4,288 count= 3 avg_used= 1,429 block_cnt= 2 chunk_cnt= 2 mod=NegParamIdex [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=TableLockNode [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 2 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSIDDLVar [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 343,733,744 used= 336,151,954 count= 31,593 avg_used= 10,640 mod=SUMMARY [2025-01-08 00:51:34.908056] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=45] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 142,606,336 used= 141,492,224 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 block_cnt= 68 chunk_cnt= 68 mod=Memstore [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:34.908076] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:34.908096] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 95,862,784 used= 90,371,200 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83,884,032 used= 75,811,934 count= 2,908 avg_used= 26,070 block_cnt= 2,908 chunk_cnt= 43 mod=SqlPhyPlan [MEMORY] hold= 4,709,056 used= 4,468,656 count= 313 avg_used= 14,276 block_cnt= 313 chunk_cnt= 41 mod=SqlPlanCache [MEMORY] hold= 1,353,472 used= 1,234,048 count= 622 avg_used= 1,984 block_cnt= 208 chunk_cnt= 39 mod=CreateContext [MEMORY] hold= 422,656 used= 412,672 count= 52 avg_used= 7,936 block_cnt= 52 chunk_cnt= 12 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 90,371,200 used= 81,928,910 count= 3,897 avg_used= 21,023 mod=SUMMARY [2025-01-08 00:51:34.908116] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,368 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,272 used= 58,751 count= 259 avg_used= 226 block_cnt= 21 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,368 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-01-08 00:51:34.908130] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 100,663,296 used= 96,509,952 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 96,509,952 used= 96,330,432 count= 187 avg_used= 515,136 block_cnt= 187 chunk_cnt= 48 mod=CoStack [MEMORY] hold= 96,509,952 used= 96,330,432 count= 187 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:34.908151] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 213,909,504 used= 213,909,504 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:34.908168] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 14,680,064 used= 12,713,920 limit= 429,496,720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,036,352 used= 7,149,419 count= 109 avg_used= 65,591 block_cnt= 109 chunk_cnt= 5 mod=L_TabletPool [MEMORY] hold= 4,587,520 used= 4,569,600 count= 70 avg_used= 65,280 block_cnt= 70 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 12,713,920 used= 11,803,787 count= 181 avg_used= 65,214 mod=SUMMARY [2025-01-08 00:51:34.908182] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,584,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 block_cnt= 195 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:34.908345] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=148] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 16,256 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ExtInfoLog [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:34.908361] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 33,216 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 mod=SUMMARY [2025-01-08 00:51:34.908386] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 14,025,664 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 9,473,984 used= 9,445,184 count= 3 avg_used= 3,148,394 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 4,543,552 used= 4,436,224 count= 559 avg_used= 7,936 block_cnt= 559 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 14,025,664 used= 13,889,344 count= 563 avg_used= 24,670 mod=SUMMARY [2025-01-08 00:51:34.908405] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 73,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 73,152 used= 71,424 count= 9 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:34.908442] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 633,344,000 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 232,894,464 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 8,388,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 24,260,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 170,676,224 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98,557,952 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:34.908424] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=31] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:34.908484] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=58] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:34.908585] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309254996000, latest_log_ts:{val:1736326292647912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:34.908670] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=81] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309254996000, latest_log_ts:{val:1736326292647912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:34.908728] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=51] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:34.908802] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=69] get rec log scn(service_type_=0, rec_log_ts={val:1736326292647912001, v:0}) [2025-01-08 00:51:34.908816] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:34.908824] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:34.908791] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 232,894,464 used= 186,015,280 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,397,376 used= 7,340,256 count= 7 avg_used= 1,048,608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7,259,504 used= 6,324,350 count= 1,032 avg_used= 6,128 block_cnt= 540 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,783,552 used= 5,752,960 count= 6 avg_used= 958,826 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 16 mod=Iteratorget_ls_id()={id:1}) [2025-01-08 00:51:34.909056] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=80] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:34.909070] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 170,676,224 used= 152,060,432 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,393,440 used= 78,764,016 count= 8,105 avg_used= 9,717 block_cnt= 1,360 chunk_cnt= 47 mod=SchemaSysCache [MEMORY] hold= 52,067,968 used= 52,041,216 count= 31 avg_used= 1,678,748 block_cnt= 31 chunk_cnt= 29 mod=TenantSchemMgr [MEMORY] hold= 7,098,368 used= 7,078,824 count= 1 avg_used= 7,078,824 block_cnt= 1 chunk_cnt= 1 mod=SchemaIdVersion [MEMORY] hold= 4,600,448 used= 4,491,776 count= 566 avg_used= 7,936 block_cnt= 566 chunk_cnt= 4 mod=SchemaGetteGuar [MEMORY] hold= 2,097,024 used= 2,095,616 count= 3 avg_used= 698,538 block_cnt= 3 chunk_cnt= 3 mod=TenaSchMgrForLi [MEMORY] hold= 1,028,544 used= 807,264 count= 74 avg_used= 10,908 block_cnt= 74 chunk_cnt= 18 mod=ScheTablInfoVec [MEMORY] hold= 720,896 used= 655,360 count= 8 avg_used= 81,920 block_cnt= 8 chunk_cnt= 2 mod=SchemaMgrCache [MEMORY] hold= 633,984 used= 619,008 count= 78 avg_used= 7,936 block_cnt= 78 chunk_cnt= 14 mod=ScheTablIdMap [MEMORY] hold= 455,168 used= 444,416 count= 56 avg_used= 7,936 block_cnt= 56 chunk_cnt= 13 mod=ScheTablNameMap [MEMORY] hold= 437,184 used= 426,816 count= 54 avg_used= 7,904 block_cnt= 54 chunk_cnt= 12 mod=ScheTenaInfoVec [MEMORY] hold= 259,072 used= 252,928 count= 32 avg_used= 7,904 block_cnt= 32 chunk_cnt= 12 mod=ScheLobMInfoVec [MEMORY] hold= 259,072 used= 252,928 count= 32 avg_used= 7,904 block_cnt= 32 chunk_cnt= 12 mod=ScheLobPInfoVec [MEMORY] hold= 235,648 used= 224,896 count= 56 avg_used= 4,016 block_cnt= 53 chunk_cnt= 12 mod=SchemaSysVariab [MEMORY] hold= 235,648 used= 224,896 count= 56 avg_used= 4,016 block_cnt= 53 chunk_cnt= 14 mod=SchemaKeystore [MEMORY] hold= 226,688 used= 221,312 count= 28 avg_used= 7,904 block_cnt= 28 chunk_cnt= 10 mod=SchePrivDbPrivs [MEMORY] hold= 226,688 used= 221,312 count= 28 avg_used= 7,904 block_cnt= 28 chunk_cnt= 13 mod=ScheDbInfoVec [MEMORY] hold= 226,688 used= 221,312 count= 28 avg_used= 7,904 block_cnt= 28 chunk_cnt= 12 mod=ScheIndeInfoVec [MEMORY] hold= 62,720 used= 57,344 count= 28 avg_used= 2,048 block_cnt= 25 chunk_cnt= 11 mod=ScheIndeNameMap [MEMORY] hold= 40,480 used= 39,520 count= 5 avg_used= 7,904 block_cnt= 5 chunk_cnt= 4 mod=SchePackInfoVec [MEMORY] hold= 40,480 used= 39,520 count= 5 avg_used= 7,904 block_cnt= 5 chunk_cnt= 4 mod=ScheRoutInfoVec [MEMORY] hold= 39,168 used= 33,792 count= 28 avg_used= 1,206 block_cnt= 26 chunk_cnt= 11 mod=ScheRoutIdMap [MEMORY] hold= 39,168 used= 33,792 count= 28 avg_used= 1,206 block_cnt= 25 chunk_cnt= 10 mod=ScheRoutNameMap [MEMORY] hold= 35,840 used= 14,336 count= 112 avg_used= 128 block_cnt= 44 chunk_cnt= 13 mod=ScheLabeSeCompo [MEMORY] hold= 34,048 used= 28,672 count= 28 avg_used= 1,024 block_cnt= 25 chunk_cnt= 11 mod=HiddenTblNames [MEMORY] hold= 34,048 used= 28,672 count= 28 avg_used= 1,024 block_cnt= 25 chunk_cnt= 11 mod=SchePackIdMap [MEMORY] hold= 34,048 used= 28,672 count= 28 avg_used= 1,024 block_cnt= 25 chunk_cnt= 9 mod=SchePackNameMap [MEMORY] hold= 34,048 used= 28,672 count= 28 avg_used= 1,024 block_cnt= 24 chunk_cnt= 11 mod=BuiltInIdxNames [MEMORY] hold= 26,880 used= 10,752 count= 84 avg_used= 128 block_cnt= 34 chunk_cnt= 11 mod=ScheLabeSeLabel [MEMORY] hold= 26,880 used= 10,752 count= 84 avg_used= 128 block_cnt= 35 chunk_cnt= 14 mod=ScheLabeSePolic [MEMORY] hold= 25,152 used= 19,392 count= 30 avg_used= 646 block_cnt= 28 chunk_cnt= 13 mod=SchemaSequence [MEMORY] hold= 19,712 used= 14,336 count= 28 avg_used= 512 block_cnt= 23 chunk_cnt= 11 mod=ScheUdtNameMap [MEMORY] hold= 19,712 used= 14,336 count= 28 avg_used= 512 block_cnt= 26 chunk_cnt= 10 mod=ScheUdtIdMap [MEMORY] hold= 17,952 used= 7,168 count= 56 avg_used= 128 block_cnt= 36 chunk_cnt= 14 mod=SchemaSynonym [MEMORY] hold= 17,920 used= 7,168 count= 56 avg_used= 128 block_cnt= 32 chunk_cnt= 12 mod=RLS_GROUP_MGR [MEMORY] hold= 17,920 used= 7,168 count= 56 avg_used= 128 block_cnt= 38 chunk_cnt= 12 mod=RLS_CONTEXT_MGR [MEMORY] hold= 17,920 used= 7,168 count= 56 avg_used= 128 block_cnt= 26 chunk_cnt= 11 mod=RLS_POLICY_MGR [MEMORY] hold= 17,920 used= 7,168 count= 56 avg_used= 128 block_cnt= 40 chunk_cnt= 13 mod=ScheOutlSqlMap [MEMORY] hold= 17,920 used= 7,168 count= 56 avg_used= 128 block_cnt= 30 chunk_cnt= 11 mod=DBLINK_MGR [MEMORY] hold= 17,920 used= 7,168 count= 56 avg_used= 128 block_cnt= 30 chunk_cnt= 10 mod=ScheLabSeUserLe [MEMORY] hold= 17,920 used= 7,168 count= 56 avg_used= 128 block_cnt= 33 chunk_cnt= 13 mod=SchemaProfile [MEMORY] hold= 17,920 used= 7,168 count= 56 avg_used= 128 block_cnt= 27 chunk_cnt= 12 mod=DIRECTORY_MGR [MEMORY] hold= 16,192 used= 15,808 count= 2 avg_used= 7,904 block_cnt= 2 chunk_cnt= 2 mod=ScheUserInfoVec [MEMORY] hold= 12,544 used= 7,168 count= 28 avg_used= 256 block_cnt= 25 chunk_cnt= 10 mod=ScheDataNameMap [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 26 chunk_cnt= 9 mod=ScheConsNameMap [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 25 chunk_cnt= 10 mod=SchePriTabPriMa [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 24 chunk_cnt= 10 mod=PRIV_ROUTINE [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 25 chunk_cnt= 11 mod=SchePriObjPriMa [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 22 chunk_cnt= 10 mod=ScheAuxVpNameVe [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 25 chunk_cnt= 8 mod=ScheForKeyNamMa [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 25 chunk_cnt= 11 mod=ScheTrigIdMap [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 25 chunk_cnt= 10 mod=ScheTrigNameMap [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 25 chunk_cnt= 10 mod=SchemaUdf [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 25 chunk_cnt= 11 mod=ScheOutlIdMap [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 27 chunk_cnt= 12 mod=SchemaSecurAudi [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 24 chunk_cnt= 11 mod=MockFkParentTab [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 27 chunk_cnt= 11 mod=SchemaContext [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 26 chunk_cnt= 11 mod=SchemaTablespac [MEMORY] hold= 8,960 used= 3,584 count= 28 avg_used= 128 block_cnt= 25 chunk_cnt= 11 mod=ScheOutlNameMap [MEMORY] hold= 8,192 used= 7,968 count= 1 avg_used= 7,968 block_cnt= 1 chunk_cnt= 1 mod=ScheMgrInfoArra [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SchemaTasks [MEMORY] hold= 2,400 used= 1,752 count= 3 avg_used= 584 block_cnt= 3 chunk_cnt= 3 mod=TenaScheMemMgr [MEMORY] hold= 2,352 used= 1,752 count= 3 avg_used= 584 block_cnt= 3 chunk_cnt= 3 mod=TenSchMemMgrFoL [MEMORY] hold= 152,060,432 used= 149,607,112 count= 10,770 avg_used= 13,891 mod=SUMMARY [2025-01-08 00:51:34.909104] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98,557,952 used= 94,211,056 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 17,440,768 used= 17,423,400 count= 1 avg_used= 17,423,400 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MB_HANDLE [MEMORY] hold= 17,338,048 used= 8,968,960 count= 1,093 avg_used= 8,205 block_cnt= 1,093 chunk_cnt= 10 mod=StorageLoggerM [MEMORY] hold= 16,818,384 used= 16,777,232 count= 3 avg_used= 5,592,410 block_cnt= 3 chunk_cnt= 3 mod=CACHE_MAP_BKT [MEMORY] hold= 16,806,528 used= 16,785,696 count= 3 avg_used= 5,595,232 block_cnt= 3 chunk_cnt= 2 mod=FixeSizeBlocAll [MEMORY] hold= 13,946,880 used= 13,894,344 count= 3 avg_used= 4,631,448 block_cnt= 3 chunk_cnt= 3 mod=TenantConfig [MEMORY] hold= 9,637,888 used= 8,551,776 count= 240 avg_used= 35,632 block_cnt= 144 chunk_cnt= 6 mod=OccamThreadPool [MEMORY] hold= 1,581,056 used= 1,572,872 count= 1 avg_used= 1,572,872 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MAP_LOCK [MEMORY] hold= 532,992 used= 430,848 count= 24 avg_used= 17,952 block_cnt= 20 chunk_cnt= 6 mod=CommonNetwork [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 2 mod=BlockMap [MEMORY] hold= 43,488 used= 3,712 count= 232 avg_used= 16 block_cnt= 8 chunk_cnt= 3 mod=ConfigChecker [MEMORY] hold= 94,211,056 used= 84,472,328 count= 1,608 avg_used= 52,532 mod=SUMMARY [2025-01-08 00:51:34.909126] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 508, limit: 1,073,741,824 hold: 15,572,992 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5,087,232 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10,485,760 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:34.909151] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5,087,232 used= 3,965,584 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 180,224 used= 88,528 count= 11 avg_used= 8,048 block_cnt= 11 chunk_cnt= 1 mod=[T]ObPerfEventR [MEMORY] hold= 52,096 used= 49,984 count= 11 avg_used= 4,544 block_cnt= 11 chunk_cnt= 1 mod=[T]ObTraceEvent [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 12,848 used= 8,960 count= 20 avg_used= 448 block_cnt= 4 chunk_cnt= 1 mod=OMT_Worker [MEMORY] hold= 10,560 used= 2,560 count= 40 avg_used= 64 block_cnt= 4 chunk_cnt= 1 mod=Coro [MEMORY] hold= 9,504 used= 7,392 count= 11 avg_used= 672 block_cnt= 11 chunk_cnt= 1 mod=[T]ObWarningBuf [MEMORY] hold= 4,480 used= 480 count= 20 avg_used= 24 block_cnt= 4 chunk_cnt= 1 mod=[T]MemoryContex [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 3,965,584 used= 3,826,232 count= 117 avg_used= 32,702 mod=SUMMARY [2025-01-08 00:51:34.909283] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10,485,760 used= 10,321,920 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10,321,920 used= 10,302,720 count= 20 avg_used= 515,136 block_cnt= 20 chunk_cnt= 5 mod=CoStack [MEMORY] hold= 10,321,920 used= 10,302,720 count= 20 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:34.909359] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant: 509, limit: 1,073,741,824 hold: 15,572,992 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5,087,232 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10,485,760 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:34.909383] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5,087,232 used= 3,723,760 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 12,848 used= 8,960 count= 20 avg_used= 448 block_cnt= 4 chunk_cnt= 1 mod=OMT_Worker [MEMORY] hold= 10,560 used= 2,560 count= 40 avg_used= 64 block_cnt= 4 chunk_cnt= 1 mod=Coro [MEMORY] hold= 4,480 used= 480 count= 20 avg_used= 24 block_cnt= 4 chunk_cnt= 1 mod=[T]MemoryContex [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 3,723,760 used= 3,680,328 count= 84 avg_used= 43,813 mod=SUMMARY [2025-01-08 00:51:34.909484] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=39] PNIO [ratelimit] time: 1736326294909482, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000771, add_bytes: 0 [2025-01-08 00:51:34.909495] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10,485,760 used= 10,321,920 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10,321,920 used= 10,302,720 count= 20 avg_used= 515,136 block_cnt= 20 chunk_cnt= 5 mod=CoStack [MEMORY] hold= 10,321,920 used= 10,302,720 count= 20 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:34.909583] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant: 1001, limit: 1,073,741,824 hold: 647,712,768 cache_hold: 79,691,776 cache_used: 79,691,776 cache_item_count: 38 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 316,342,272 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 102,760,448 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 46,137,344 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 62,914,560 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 6,291,456 limit= 214,748,360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 23,089,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:34.909885] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 316,342,272 used= 253,839,984 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 15 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 18,276,352 used= 18,132,544 count= 20 avg_used= 906,627 block_cnt= 20 chunk_cnt= 14 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,646,144 used= 16,637,952 count= 8 avg_used= 2,079,744 block_cnt= 8 chunk_cnt= 8 mod=CACHE_MAP_NODE [MEMORY] hold= 16,196,944 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 41 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 4,214,784 used= 4,194,336 count= 1 avg_used= 4,194,336 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 16 mod=CkptDgnMemCU [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 17 mod=CkptDgnMem [MEMORY] hold= 3,809,280 used= 3,770,880 count= 5 avg_used= 754,176 block_cnt= 5 chunk_cnt= 4 mod=ResourceGroup [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,686,400 used= 3,211,264 count= 58 avg_used= 55,366 block_cnt= 58 chunk_cnt= 16 mod=[T]char [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,933,312 used= 1,914,880 count= 3 avg_used= 638,293 block_cnt= 3 chunk_cnt= 2 mod=ServerObjecPool [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,337,616 used= 1,316,664 count= 68 avg_used= 19,362 block_cnt= 68 chunk_cnt= 10 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,092,032 used= 1,046,832 count= 226 avg_used= 4,632 block_cnt= 226 chunk_cnt= 16 mod=WorkerMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=DiagnosticInfos [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 917,504 used= 870,400 count= 8 avg_used= 108,800 block_cnt= 8 chunk_cnt= 6 mod=DiagnosticSum [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 3 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 458,176 used= 440,550 count= 6 avg_used= 73,425 block_cnt= 6 chunk_cnt= 4 mod=SqlMemMgr [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 262,144 used= 128,768 count= 16 avg_used= 8,048 block_cnt= 16 chunk_cnt= 9 mod=[T]ObPerfEventR [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 196,608 used= 195,840 count= 3 avg_used= 65,280 block_cnt= 3 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 187,200 used= 178,560 count= 45 avg_used= 3,968 block_cnt= 45 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 180,224 used= 103,312 count= 11 avg_used= 9,392 block_cnt= 11 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,456 used= 146,976 count= 1 avg_used= 146,976 block_cnt= 1 chunk_cnt= 1 mod=SQLSessionInfo [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 7 mod=QueryAllocator [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 8 mod=LinearHashMapNo [MEMORY] hold= 89,408 used= 87,296 count= 11 avg_used= 7,936 block_cnt= 11 chunk_cnt= 5 mod=ParamCharsType [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 7 mod=TabletIDSetNode [MEMORY] hold= 75,776 used= 72,704 count= 16 avg_used= 4,544 block_cnt= 16 chunk_cnt= 9 mod=[T]ObTraceEvent [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 60,432 used= 15,008 count= 226 avg_used= 66 block_cnt= 58 chunk_cnt= 23 mod=Coro [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=MdsTableMgr [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 3 mod=HashNodeLCSta [MEMORY] hold= 29,184 used= 27,648 count= 8 avg_used= 3,456 block_cnt= 8 chunk_cnt= 6 mod=CACHE_INST [MEMORY] hold= 28,352 used= 23,232 count= 2 avg_used= 11,616 block_cnt= 2 chunk_cnt= 2 mod=DI_CONTAINER [MEMORY] hold= 25,952 used= 2,760 count= 115 avg_used= 24 block_cnt= 43 chunk_cnt= 16 mod=[T]MemoryContex [MEMORY] hold= 25,552 used= 19,232 count= 2 avg_used= 9,616 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 24,576 used= 21,536 count= 1 avg_used= 21,536 block_cnt= 1 chunk_cnt= 1 mod=DiagnosticInfo [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 9,336 count= 1 avg_used= 9,336 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_AssignedMgr [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,000 used= 15,616 count= 2 avg_used= 7,808 block_cnt= 2 chunk_cnt= 2 mod=HashNodePlanCac [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 15,360 used= 10,752 count= 24 avg_used= 448 block_cnt= 13 chunk_cnt= 6 mod=OMT_Worker [MEMORY] hold= 14,688 used= 11,424 count= 17 avg_used= 672 block_cnt= 16 chunk_cnt= 9 mod=[T]ObWarningBuf [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 2 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PartCtx [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,880 used= 1,920 count= 5 avg_used= 384 block_cnt= 5 chunk_cnt= 3 mod=TableLockNode [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,496 used= 1,920 count= 3 avg_used= 640 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 2 chunk_cnt= 2 mod=DICollector [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 1 chunk_cnt= 1 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 288 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 253,839,984 used= 250,037,127 count= 3,091 avg_used= 80,891 mod=SUMMARY [2025-01-08 00:51:34.909936] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=39] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 102,760,448 used= 101,957,632 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 101,957,632 used= 101,910,984 count= 49 avg_used= 2,079,816 block_cnt= 49 chunk_cnt= 49 mod=Memstore [MEMORY] hold= 101,957,632 used= 101,910,984 count= 49 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:34.909956] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:34.909976] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 46,137,344 used= 30,700,864 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 28,643,136 used= 25,577,143 count= 1,233 avg_used= 20,743 block_cnt= 1,233 chunk_cnt= 22 mod=SqlPhyPlan [MEMORY] hold= 1,172,416 used= 1,085,952 count= 125 avg_used= 8,687 block_cnt= 125 chunk_cnt= 19 mod=SqlPlanCache [MEMORY] hold= 574,464 used= 523,776 count= 264 avg_used= 1,984 block_cnt= 88 chunk_cnt= 18 mod=CreateContext [MEMORY] hold= 308,864 used= 301,568 count= 38 avg_used= 7,936 block_cnt= 38 chunk_cnt= 7 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 30,700,864 used= 27,490,039 count= 1,662 avg_used= 16,540 mod=SUMMARY [2025-01-08 00:51:34.909998] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 62,914,560 used= 60,383,232 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 60,383,232 used= 60,270,912 count= 117 avg_used= 515,136 block_cnt= 117 chunk_cnt= 30 mod=CoStack [MEMORY] hold= 60,383,232 used= 60,270,912 count= 117 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:34.910051] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 79,691,776 used= 79,691,776 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 79,691,776 used= 0 count= 38 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 79,691,776 used= 0 count= 38 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:34.910068] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 6,291,456 used= 4,153,280 limit= 214,748,360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4,063,232 used= 4,047,360 count= 62 avg_used= 65,280 block_cnt= 62 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 4,153,280 used= 4,132,128 count= 64 avg_used= 64,564 mod=SUMMARY [2025-01-08 00:51:34.910082] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,064,768 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,064,768 used= 1,039,616 count= 131 avg_used= 7,936 block_cnt= 131 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,064,768 used= 1,039,616 count= 131 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:34.910094] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:34.910139] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 0 used= 49,152 limit= 1,073,741,800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 49,152 used= 32,512 count= 2 avg_used= 16,256 block_cnt= 2 chunk_cnt= 1 mod=OB_LOG_PUSH_REQ [MEMORY] hold= 49,152 used= 32,512 count= 2 avg_used= 16,256 mod=SUMMARY [2025-01-08 00:51:34.910161] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 23,089,152 used= 22,853,184 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 17,862,592 used= 17,833,792 count= 3 avg_used= 5,944,597 block_cnt= 3 chunk_cnt= 2 mod=MEMTABLE_ARENA [MEMORY] hold= 4,982,464 used= 4,864,768 count= 613 avg_used= 7,936 block_cnt= 613 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 22,853,184 used= 22,706,496 count= 617 avg_used= 36,801 mod=SUMMARY [2025-01-08 00:51:34.910234] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 4,194,304 used= 384,320 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 359,936 used= 351,088 count= 10 avg_used= 35,108 block_cnt= 10 chunk_cnt= 2 mod=TX_OP [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 384,320 used= 374,896 count= 13 avg_used= 28,838 mod=SUMMARY [2025-01-08 00:51:34.910279] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant: 1002, limit: 3,221,225,472 hold: 1,114,390,528 cache_hold: 52,428,800 cache_used: 52,428,800 cache_item_count: 25 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 596,312,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 123,731,968 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 102,760,448 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 23,068,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 98,566,144 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 4,194,304 limit= 644,245,080 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 69,287,936 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33,554,432 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:34.910597] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 596,312,064 used= 507,008,896 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 30 [MEMORY] hold= 167,854,080 used= 167,772,160 count= 4 avg_used= 41,943,040 block_cnt= 4 chunk_cnt= 4 mod=LogGroupBuffer [MEMORY] hold= 82,780,160 used= 82,604,608 count= 51 avg_used= 1,619,698 block_cnt= 51 chunk_cnt= 43 mod=MysqlRequesReco [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 7 mod=IoControl [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,646,144 used= 16,637,952 count= 8 avg_used= 2,079,744 block_cnt= 8 chunk_cnt= 8 mod=CACHE_MAP_NODE [MEMORY] hold= 16,196,944 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 41 chunk_cnt= 7 mod=OmtTenant [MEMORY] hold= 14,893,056 used= 14,794,816 count= 8 avg_used= 1,849,352 block_cnt= 8 chunk_cnt= 8 mod=LogDIOAligned [MEMORY] hold= 13,975,552 used= 13,926,400 count= 3 avg_used= 4,642,133 block_cnt= 3 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 6,856,704 used= 6,787,584 count= 9 avg_used= 754,176 block_cnt= 9 chunk_cnt= 9 mod=ResourceGroup [MEMORY] hold= 6,340,608 used= 5,505,024 count= 102 avg_used= 53,970 block_cnt= 102 chunk_cnt= 19 mod=[T]char [MEMORY] hold= 4,943,872 used= 4,915,456 count= 2 avg_used= 2,457,728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 20 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 20 mod=CkptDgnMemCU [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 6 mod=SqlDtlQueue [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,831,808 used= 2,713,104 count= 594 avg_used= 4,567 block_cnt= 583 chunk_cnt= 21 mod=WorkerMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,326,528 used= 2,284,800 count= 11 avg_used= 207,709 block_cnt= 11 chunk_cnt= 9 mod=DiagnosticSum [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,978,176 used= 1,960,616 count= 50 avg_used= 39,212 block_cnt= 50 chunk_cnt= 7 mod=TabletMap [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,605,632 used= 1,574,016 count= 4 avg_used= 393,504 block_cnt= 4 chunk_cnt= 4 mod=ClogGe [MEMORY] hold= 1,376,256 used= 1,343,488 count= 4 avg_used= 335,872 block_cnt= 4 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 1,335,296 used= 1,331,072 count= 1 avg_used= 1,331,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=DiagnosticInfos [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 870,848 used= 849,152 count= 96 avg_used= 8,845 block_cnt= 96 chunk_cnt= 16 mod=ParamCharsType [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 3 chunk_cnt= 3 mod=PartitTableTask [MEMORY] hold= 787,264 used= 524,352 count= 36 avg_used= 14,565 block_cnt= 34 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 720,896 used= 354,112 count= 44 avg_used= 8,048 block_cnt= 44 chunk_cnt= 17 mod=[T]ObPerfEventR [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 589,824 used= 587,904 count= 4 avg_used= 146,976 block_cnt= 4 chunk_cnt= 3 mod=SQLSessionInfo [MEMORY] hold= 491,520 used= 442,432 count= 8 avg_used= 55,304 block_cnt= 8 chunk_cnt= 5 mod=TabletIDSetBkt [MEMORY] hold= 466,880 used= 451,312 count= 3 avg_used= 150,437 block_cnt= 3 chunk_cnt= 3 mod=IndexUsageMgr [MEMORY] hold= 458,752 used= 444,384 count= 4 avg_used= 111,096 block_cnt= 4 chunk_cnt= 3 mod=AsyncFreezeTabl [MEMORY] hold= 450,272 used= 432,838 count= 5 avg_used= 86,567 block_cnt= 5 chunk_cnt= 4 mod=SqlMemMgr [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 393,216 used= 372,096 count= 4 avg_used= 93,024 block_cnt= 4 chunk_cnt= 4 mod=LSSvr [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 1 mod=bf_queue [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 278,528 used= 159,664 count= 17 avg_used= 9,392 block_cnt= 17 chunk_cnt= 9 mod=[T]ObDSActionAr [MEMORY] hold= 270,336 used= 246,624 count= 3 avg_used= 82,208 block_cnt= 3 chunk_cnt= 2 mod=HTblRkQueAlloc [MEMORY] hold= 229,376 used= 221,688 count= 1 avg_used= 221,688 block_cnt= 1 chunk_cnt= 1 mod=IndexUsageRepor [MEMORY] hold= 229,376 used= 203,776 count= 4 avg_used= 50,944 block_cnt= 4 chunk_cnt= 3 mod=palf_env [MEMORY] hold= 213,120 used= 204,480 count= 45 avg_used= 4,544 block_cnt= 45 chunk_cnt= 17 mod=[T]ObTraceEvent [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 163,840 used= 155,664 count= 2 avg_used= 77,832 block_cnt= 2 chunk_cnt= 2 mod=IteratorStorage [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 137,280 used= 130,944 count= 33 avg_used= 3,968 block_cnt= 33 chunk_cnt= 7 mod=TinyAllocator [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=LockMemObj [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98,304 used= 86,144 count= 4 avg_used= 21,536 block_cnt= 4 chunk_cnt= 2 mod=DiagnosticInfo [MEMORY] hold= 91,264 used= 23,584 count= 336 avg_used= 70 block_cnt= 83 chunk_cnt= 26 mod=Coro [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81,280 used= 79,360 count= 10 avg_used= 7,936 block_cnt= 10 chunk_cnt= 6 mod=QueryAllocator [MEMORY] hold= 81,280 used= 79,360 count= 10 avg_used= 7,936 block_cnt= 10 chunk_cnt= 8 mod=LinearHashMapNo [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=MinorMergeMgr [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 46,336 count= 4 avg_used= 11,584 block_cnt= 4 chunk_cnt= 4 mod=LogApplyStatus [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 40,992 count= 4 avg_used= 10,248 block_cnt= 4 chunk_cnt= 4 mod=TabletSvrBucket [MEMORY] hold= 65,536 used= 56,064 count= 4 avg_used= 14,016 block_cnt= 4 chunk_cnt= 3 mod=LogReplayStatus [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 6 mod=HashNodeLCSta [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 41,664 used= 4,416 count= 184 avg_used= 24 block_cnt= 61 chunk_cnt= 24 mod=[T]MemoryContex [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 38,400 used= 26,880 count= 60 avg_used= 448 block_cnt= 32 chunk_cnt= 15 mod=OMT_Worker [MEMORY] hold= 38,016 used= 29,568 count= 44 avg_used= 672 block_cnt= 43 chunk_cnt= 17 mod=[T]ObWarningBuf [MEMORY] hold= 32,768 used= 31,744 count= 2 avg_used= 15,872 block_cnt= 2 chunk_cnt= 2 mod=NotParamInfo [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MatchLsnMap [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 31,936 used= 26,816 count= 2 avg_used= 13,408 block_cnt= 2 chunk_cnt= 2 mod=DI_CONTAINER [MEMORY] hold= 29,184 used= 27,648 count= 8 avg_used= 3,456 block_cnt= 5 chunk_cnt= 5 mod=CACHE_INST [MEMORY] hold= 28,992 used= 27,792 count= 6 avg_used= 4,632 block_cnt= 6 chunk_cnt= 3 mod=TTLTaskBucket [MEMORY] hold= 25,728 used= 24,576 count= 6 avg_used= 4,096 block_cnt= 6 chunk_cnt= 5 mod=LinearHashMapCn [MEMORY] hold= 25,728 used= 24,576 count= 6 avg_used= 4,096 block_cnt= 6 chunk_cnt= 6 mod=LinearHashMapDi [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=PartCtx [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ObDataDictSvc [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ALLTTLTaskSche [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,616 used= 7,424 count= 1 avg_used= 7,424 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,976 used= 6,176 count= 4 avg_used= 1,544 block_cnt= 4 chunk_cnt= 3 mod=TXBFRetryId [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 3,968 used= 3,200 count= 4 avg_used= 800 block_cnt= 4 chunk_cnt= 3 mod=DICollector [MEMORY] hold= 3,904 used= 3,104 count= 4 avg_used= 776 block_cnt= 4 chunk_cnt= 3 mod=DepTabletSet [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 3,136 used= 2,560 count= 3 avg_used= 853 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 2,752 used= 2,176 count= 3 avg_used= 725 block_cnt= 3 chunk_cnt= 3 mod=NotParamIdex [MEMORY] hold= 2,688 used= 1,888 count= 4 avg_used= 472 block_cnt= 4 chunk_cnt= 3 mod=Election [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,304 used= 1,344 count= 5 avg_used= 268 block_cnt= 4 chunk_cnt= 4 mod=NegParamIdex [MEMORY] hold= 2,016 used= 864 count= 6 avg_used= 144 block_cnt= 5 chunk_cnt= 5 mod=ObTxLSLogCb [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,888 used= 256 count= 8 avg_used= 32 block_cnt= 6 chunk_cnt= 4 mod=PalfFSCbNode [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,216 used= 416 count= 4 avg_used= 104 block_cnt= 4 chunk_cnt= 3 mod=RedoLogBuckLock [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 960 used= 164 count= 4 avg_used= 41 block_cnt= 4 chunk_cnt= 4 mod=KeepAliveBuf [MEMORY] hold= 896 used= 128 count= 4 avg_used= 32 block_cnt= 3 chunk_cnt= 3 mod=PalfRCCbNode [MEMORY] hold= 896 used= 128 count= 4 avg_used= 32 block_cnt= 4 chunk_cnt= 4 mod=RebuildCbNode [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 640 used= 440 count= 1 avg_used= 440 block_cnt= 1 chunk_cnt= 1 mod=tenant_ttl_mgr [MEMORY] hold= 608 used= 204 count= 2 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=TableLockNode [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 507,008,896 used= 501,868,136 count= 4,025 avg_used= 124,687 mod=SUMMARY [2025-01-08 00:51:34.910674] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=59] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 123,731,968 used= 122,765,312 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 122,765,312 used= 122,709,144 count= 59 avg_used= 2,079,816 block_cnt= 59 chunk_cnt= 59 mod=Memstore [MEMORY] hold= 122,765,312 used= 122,709,144 count= 59 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:34.910696] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 4,194,304 used= 2,916,352 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,916,352 used= 2,890,240 count= 4 avg_used= 722,560 block_cnt= 4 chunk_cnt= 2 mod=PartTranCtxMgr [MEMORY] hold= 2,916,352 used= 2,890,240 count= 4 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:34.910718] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 102,760,448 used= 99,366,720 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 91,521,024 used= 82,598,964 count= 3,688 avg_used= 22,396 block_cnt= 3,688 chunk_cnt= 49 mod=SqlPhyPlan [MEMORY] hold= 4,355,904 used= 4,089,488 count= 336 avg_used= 12,171 block_cnt= 336 chunk_cnt= 47 mod=SqlPlanCache [MEMORY] hold= 1,475,136 used= 1,369,760 count= 155 avg_used= 8,837 block_cnt= 155 chunk_cnt= 2 mod=SqlPhyPlObj [MEMORY] hold= 1,427,456 used= 1,301,504 count= 656 avg_used= 1,984 block_cnt= 219 chunk_cnt= 48 mod=CreateContext [MEMORY] hold= 585,216 used= 571,392 count= 72 avg_used= 7,936 block_cnt= 72 chunk_cnt= 11 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 99,366,720 used= 89,932,708 count= 4,909 avg_used= 18,319 mod=SUMMARY [2025-01-08 00:51:34.910738] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 23,068,672 used= 22,016,304 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 22,002,832 used= 19,331,822 count= 32,760 avg_used= 590 block_cnt= 1,482 chunk_cnt= 11 mod=PlJit [MEMORY] hold= 13,472 used= 13,344 count= 2 avg_used= 6,672 block_cnt= 2 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 22,016,304 used= 19,345,166 count= 32,762 avg_used= 590 mod=SUMMARY [2025-01-08 00:51:34.910752] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 98,566,144 used= 94,961,664 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 94,961,664 used= 94,785,024 count= 184 avg_used= 515,136 block_cnt= 184 chunk_cnt= 47 mod=CoStack [MEMORY] hold= 94,961,664 used= 94,785,024 count= 184 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:34.910739] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39C5-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:34.910762] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39C5-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13300, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:34.910773] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 52,428,800 used= 52,428,800 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 52,428,800 used= 0 count= 25 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 52,428,800 used= 0 count= 25 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:34.910771] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39C5-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=44, avg_log_submit_to_flush_time=2810, avg_log_submit_to_slide_time=13255) [2025-01-08 00:51:34.910790] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 4,194,304 used= 3,178,432 limit= 644,245,080 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,014,656 used= 3,002,880 count= 46 avg_used= 65,280 block_cnt= 46 chunk_cnt= 2 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 73,728 used= 65,591 count= 1 avg_used= 65,591 block_cnt= 1 chunk_cnt= 1 mod=L_TabletPool [MEMORY] hold= 3,178,432 used= 3,153,239 count= 49 avg_used= 64,351 mod=SUMMARY [2025-01-08 00:51:34.910804] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 755,904 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 755,904 used= 738,048 count= 93 avg_used= 7,936 block_cnt= 93 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 755,904 used= 738,048 count= 93 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:34.910818] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:34.910844] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 69,287,936 used= 63,614,464 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 63,061,760 used= 62,946,560 count= 12 avg_used= 5,245,546 block_cnt= 12 chunk_cnt= 9 mod=MEMTABLE_ARENA [MEMORY] hold= 520,192 used= 507,904 count= 64 avg_used= 7,936 block_cnt= 64 chunk_cnt= 1 mod=TX_DATA_SLICE [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 63,614,464 used= 63,486,208 count= 80 avg_used= 793,577 mod=SUMMARY [2025-01-08 00:51:34.910864] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 65,024 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:34.910885] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33,554,432 used= 0 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 33,554,432 free_size= 33,554,432 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-01-08 00:51:34.911053] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:34.914293] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43401313}], scn_array=[cnt:1, {val:1736326294887400000, v:0}], curr_size=546, accum_size=8745, time_cost=14852) [2025-01-08 00:51:34.914464] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=51] [APPLY STAT CB APPEND COST TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=120, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:34.914485] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=21] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=4673, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:34.914493] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:34.914503] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB EXECUTE TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=-884, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:34.914529] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=25] [APPLY STAT CB TOTAL TIME](cur_stat_count=59, stat_interval=5000000, avg (count/cost)=3909, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:34.924827] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2212036}], scn_array=[cnt:1, {val:1736326294817229000, v:0}], curr_size=122, accum_size=723, time_cost=13966) [2025-01-08 00:51:34.933843] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:34.939284] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE2-0-0] [lt=26] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:34.939317] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE2-0-0] [lt=26] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=36) [2025-01-08 00:51:34.942379] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:34.943859] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1038081, clean_num=31457) [2025-01-08 00:51:34.958751] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=24] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12803338240, fast_delete_log_mode=false) [2025-01-08 00:51:34.971891] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=20] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:34.972011] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=102][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:34.989281] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C691-0-0] [lt=52][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193799903112, cache_obj->added_lc()=false, cache_obj->get_object_id()=13652, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:34.991501] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B5-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:34.991546] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B5-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326294828535000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326294828535000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:35.010840] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39C7-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18174, log_task={header:{begin_lsn:{lsn:2212036}, end_lsn:{lsn:2212158}, log_id:18174, min_scn:{val:1736326294817229000, v:0}, max_scn:{val:1736326294817229000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2211914}, prev_proposal_id:1, committed_end_lsn:{lsn:2212036}, data_checksum:940009244, accum_checksum:2151623894, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326294910616, freeze_ts:1736326294910616, submit_ts:1736326294910630, flushed_ts:1736326294925055, gen_to_freeze cost time:0, gen_to_submit cost time:14, submit_to_flush cost time:14425}, fs_cb_begin_ts=1736326295010828, log_life_time=100212) [2025-01-08 00:51:35.012556] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] succ to submit log task to replay service(to_submit_lsn={lsn:2212092}, to_submit_scn={val:1736326294817229000, v:0}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326295010832, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2212158}, next_to_submit_scn_:{val:1736326294817229001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2212036}, end_lsn:{lsn:2212158}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2212158}, readable_log_tail:{lsn:2212158}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64892610, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326294817229000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326294817229000, v:0}, data_checksum:2124447694, flag:25465}}, init_mode_version:0, accumulate_checksum:2151623894, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326294817229000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18084, miss_cnt:0, cache_read_size:2212158, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:35.015402] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CAD-0-0] [lt=56][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:35.016535] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:320K/73:312392 disk:320K/73:312392 t1m2:48K/11:266944 t1m2g0:48K/11:266944 t1002m2:188K/44:114227 t1002m2g0:188K/44:114227 t1001m2:84K/18:824568 t1001m2g0:84K/18:824568 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:35.017121] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=116, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:35.017145] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:35.017318] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CAD-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:35.019305] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43402448}], scn_array=[cnt:1, {val:1736326295004769000, v:0}], curr_size=750, accum_size=10084, time_cost=13346) [2025-01-08 00:51:35.025443] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BAD-0-0] [lt=23] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:35.036682] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=98, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:35.036731] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:51:35.036753] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:35.036768] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:35.037024] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:35.037062] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:35.037125] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:35.037141] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:35.038728] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1643) [2025-01-08 00:51:35.039564] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:35.047568] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:35.047718] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:35.048400] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58556][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=22] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:51:35.051146] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58524][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A8E-0-0] [lt=54] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11851) [2025-01-08 00:51:35.051373] INFO [LIB] stat (utility.h:1210) [58524][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A8E-0-0] [lt=108] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=14, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:35.051448] INFO [LIB] stat (utility.h:1210) [58524][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A8E-0-0] [lt=74] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4819, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:35.052319] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=109] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326294317339004, v:0}, new_max_version={val:1736326294317339004, v:0}, last_min_version={val:1736326294317339002, v:0}, last_max_version={val:1736326294317339002, v:0}, total_time_us=7279, delta_us=734976) [2025-01-08 00:51:35.052379] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=54] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326294317339004, v:0}, min_version={val:1736326294317339004, v:0}, max_version={val:1736326294317339004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:35.058092] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:35.059292] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58527][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=63] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:51:35.072122] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.134019] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:35.140097] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87D4-0-0] [lt=55] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:35.140125] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87D4-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3643, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:35.144394] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:35.149780] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326295147067000, start_id=1736326295147067000, end_id=1736326295147067001) [2025-01-08 00:51:35.151574] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=35] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=11262) [2025-01-08 00:51:35.168645] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:32940, max_lsn:43404376, max_scn:{val:1736326295147067000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:43404376}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:1461336}}, last_submit_lsn:{lsn:43403787}, last_submit_end_lsn:{lsn:43404376}, last_submit_log_id:32940, last_submit_log_pid:1, max_flushed_lsn:{lsn:43403787}, max_flushed_end_lsn:{lsn:43404376}, max_flushed_log_pid:1, committed_end_lsn:{lsn:43404376}, last_slide_log_id:32940, last_slide_scn:{val:1736326295147067000, v:0}, last_slide_lsn:{lsn:43403787}, last_slide_end_lsn:{lsn:43404376}, last_slide_log_pid:1, last_slide_log_accum_checksum:4160729841, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad9083ee530}, [StateMgr]={this:0x2ad9083f2830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:32941, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324348477669, palf_role_change_cb:0x2ad9083f92f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad9083efc70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324348678844, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/meta", dir_fd:172, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad9083f4a38, flashback_version:0}, log_storage:{log_tail:{lsn:43404376}, readable_log_tail:{lsn:43404376}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23700392, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324348477523, this:0x2ad9083f2f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.138:2882";"192.168.159.139:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324348478003, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9083f29b0}) [2025-01-08 00:51:35.169132] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=479] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326295151618, last_flushed_end_lsn:{lsn:43404376}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326295051140, last_flushed_end_lsn:{lsn:43403787}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326295153364, last_flushed_end_lsn:{lsn:43404376}}]) [2025-01-08 00:51:35.169161] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=25] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=32941, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326295151618, last_flushed_end_lsn:{lsn:43404376}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326295051140, last_flushed_end_lsn:{lsn:43403787}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326295153364, last_flushed_end_lsn:{lsn:43404376}}]) [2025-01-08 00:51:35.172618] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.192087] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=36] ====== check clog disk timer task ====== [2025-01-08 00:51:35.192142] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:35.224266] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4729-0-0] [lt=78] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:35.238788] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2384, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:35.238824] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:35.238845] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=167, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:35.238913] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=38, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:35.240011] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87D8-0-0] [lt=39] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:35.240036] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87D8-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3446, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:35.240157] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:35.247835] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=75, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:35.247870] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=11262, avg_group_log_size=625) [2025-01-08 00:51:35.247887] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11262) [2025-01-08 00:51:35.249612] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=2748, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:35.249640] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=307, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:35.249667] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=297] [PALF STAT FLUSH CB COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=13, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:35.250406] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2DF8-0-0] [lt=40] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=99, avg_log_submit_to_flush_time=3084, avg_log_submit_to_slide_time=4322) [2025-01-08 00:51:35.250583] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=11851, hit_cnt=19, read_cnt=19, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:35.265698] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=149] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1544, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:35.265756] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=213, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:35.265837] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=125] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=45, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:35.272698] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.282144] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=111] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:35.283734] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C693-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193800197566, cache_obj->added_lc()=false, cache_obj->get_object_id()=13653, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:35.284028] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=48] ====== check clog disk timer task ====== [2025-01-08 00:51:35.284102] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:35.285368] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853E2-0-0] [lt=87] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:35.285401] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853E2-0-0] [lt=25] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:35.311382] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39CD-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:35.311464] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39CD-0-0] [lt=69] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:35.311486] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39CD-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:35.312778] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2443, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:35.312810] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=343, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:35.312898] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=85, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:35.318811] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8D1-0-0] [lt=9] get number(ret=0, service_type_=0, range=1, base_id=1736326295317734000, start_id=1736326295317734000, end_id=1736326295317734001) [2025-01-08 00:51:35.319431] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.319843] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.324408] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A472B-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:35.324442] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A472B-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:35.324487] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A472B-0-0] [lt=14][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18279, log_task={header:{begin_lsn:{lsn:2224810}, end_lsn:{lsn:2224932}, log_id:18279, min_scn:{val:1736326294828535003, v:0}, max_scn:{val:1736326294828535003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2224688}, prev_proposal_id:1, committed_end_lsn:{lsn:2224810}, data_checksum:1715074492, accum_checksum:180862281, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326295224228, freeze_ts:1736326295224228, submit_ts:1736326295224236, flushed_ts:1736326295225907, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:1671}, fs_cb_begin_ts=1736326295324462, log_life_time=100234) [2025-01-08 00:51:35.326361] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:35.326889] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87D9-0-0] [lt=77] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:35.326911] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87D9-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12861, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:35.326921] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87D9-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=1456, avg_log_submit_to_slide_time=12847) [2025-01-08 00:51:35.327018] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:35.327060] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18188, miss_cnt:0, cache_read_size:2225054, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:35.330439] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1829F566-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1736326295330048000, start_id=1736326295330048000, end_id=1736326295330048001) [2025-01-08 00:51:35.334140] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:35.334941] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:35.334976] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-01-08 00:51:35.334987] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:35.335000] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:35.335014] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193799248654, flush_io_finished_ret_:0, flush_io_finished_round_:1958, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:35.338759] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1683, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:35.338827] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:35.338849] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=112, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:35.338938] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=71] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:35.339825] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] Garbage Collector is running(seq_=380, gc_interval=10000000) [2025-01-08 00:51:35.339888] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=37] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:35.339910] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=13] gc_check_member_list_ cost time(ret=0, time_us=47) [2025-01-08 00:51:35.340006] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:35.340026] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=13] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:35.340039] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:35.340077] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=36] execute_gc cost time(ret=0, time_us=38) [2025-01-08 00:51:35.340085] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:51:35.340092] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-01-08 00:51:35.340109] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-01-08 00:51:35.340720] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9410-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=1894, avg_log_submit_to_slide_time=3366) [2025-01-08 00:51:35.340837] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=89] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1643, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:35.344875] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:35.356033] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:172) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=53] Eliminate task evict sql audit(request_manager_->get_tenant_id()=1, queue_size=524288, config_mem_limit_=64424509, request_manager_->get_size_used()=0, evict_high_size_level=471859, evict_low_size_level=419430, allocator->allocated()=12478464, evict_high_mem_level=32212254, evict_low_mem_level=0) [2025-01-08 00:51:35.356069] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=12478464) [2025-01-08 00:51:35.357832] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D47-0-0] [lt=41] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:35.359347] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=52] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:35.359495] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1736326294956664001, v:0}, base_version={val:1736326294956664000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326295159569000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326295359477, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326295159569000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326295354721, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326294956664001, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326295456158, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:35.362295] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=38] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:35.362608] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=92] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:35.362674] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:35.362797] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:35.372899] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=71][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.380095] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=65] current monitor number(seq_=-1) [2025-01-08 00:51:35.400725] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC5-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:35.400765] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC5-0-0] [lt=37] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:35.406941] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=60] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:35.411636] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=42] PNIO [ratelimit] time: 1736326295411632, bytes: 878858519, bw: 0.196576 MB/s, add_ts: 1004501, add_bytes: 207053 [2025-01-08 00:51:35.412992] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1333) [2025-01-08 00:51:35.425918] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1329, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:35.451054] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=31] ====== tenant freeze timer task ====== [2025-01-08 00:51:35.458892] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=43] gc stale ls task succ [2025-01-08 00:51:35.459373] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.459428] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.459797] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:35.459845] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=42] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71124160, tenant_id=1001) [2025-01-08 00:51:35.459871] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:35.464828] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=71] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4083) [2025-01-08 00:51:35.464858] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:35.466108] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4083) [2025-01-08 00:51:35.466240] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4083) [2025-01-08 00:51:35.467742] INFO [LIB] stat (utility.h:1210) [58559][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C84-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:35.467776] INFO [LIB] stat (utility.h:1210) [58559][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C84-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3221, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:35.467789] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58559][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C84-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=123, avg_log_submit_to_flush_time=1765, avg_log_submit_to_slide_time=3098) [2025-01-08 00:51:35.467988] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4083, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:35.473062] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.519697] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=51] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.525990] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:35.526029] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=99, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:35.526101] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=58, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:35.534346] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:35.542426] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=81] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.544163] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.545416] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:35.561888] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=98, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:35.573229] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.635973] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60C-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:35.636072] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60C-0-0] [lt=81] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326295317734002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326295317734002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:35.653126] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=43410827, this={dio_aligned_buf:{buf_write_offset:2008, buf_padding_size:2088, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:45940, truncate_used_ts:13464}, log_block_size:67108864, io_fd:{first_id:-1, second_id:160, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=43409408, buf_len=589, total_write_size=96674, total_write_size_after_dio=745472, ob_pwrite_used_ts=480355, count=158, each_pwrite_cost=3040) [2025-01-08 00:51:35.659500] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:35.661490] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:35.661538] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=38] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:35.661556] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=90) [2025-01-08 00:51:35.666165] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46508][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BED0-0-0] [lt=40] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:35.672831] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485396-0-0] [lt=16] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:35.672876] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D485396-0-0] [lt=38] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:35.673621] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.673825] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BAF-0-0] [lt=13] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:35.674346] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BB1-0-0] [lt=17] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:35.692725] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [46340][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=29] XA scheduler heartbeat task statistics(avg_time=1) [2025-01-08 00:51:35.702283] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8736, offset=2480654}, end_cursor:ObLogCursor{file_id=1, log_id=8737, offset=2480959}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b74050, len:1522, log_data_len:305, seq:8736, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:35.705560] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E63-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:35.705611] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E63-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326295325887003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326295325887003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:35.709009] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E63-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1736326295705377000, start_id=1736326295705377000, end_id=1736326295705377001) [2025-01-08 00:51:35.713227] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=54] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8737, offset=2480959}, end_cursor:ObLogCursor{file_id=1, log_id=8738, offset=2481256}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b78050, len:1217, log_data_len:297, seq:8737, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:35.714425] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E63-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:35.714485] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E63-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326295325887003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326295325887003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:35.718280] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18092, miss_cnt:0, cache_read_size:2213134, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:35.719394] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=41] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:35.719420] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:35.719462] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498B-0-0] [lt=14] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:35.719498] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498B-0-0] [lt=32] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:35.721895] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=49] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8738, offset=2481256}, end_cursor:ObLogCursor{file_id=1, log_id=8739, offset=2481553}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b7c050, len:920, log_data_len:297, seq:8738, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:35.722870] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8739, offset=2481553}, end_cursor:ObLogCursor{file_id=1, log_id=8740, offset=2481850}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b80050, len:623, log_data_len:297, seq:8739, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:35.734548] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=80] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:35.738001] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:35.738081] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44932, [delay/us]:prepare:22, schedule:306, submit:185, rt:889, total:1402) [2025-01-08 00:51:35.738100] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:35.738157] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44937, [delay/us]: prepare:22, schedule:306, submit:185, device:889, total:1402) [2025-01-08 00:51:35.738170] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:35.738286] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=83] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4864, iops:16, bw:77613, [delay/us]:prepare:7, schedule:827, submit:255, rt:2008, total:3098) [2025-01-08 00:51:35.738322] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:35.738367] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4864, avg_iops:16, avg_bw:77610, [delay/us]: prepare:7, schedule:827, submit:255, device:2008, total:3098) [2025-01-08 00:51:35.738378] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:35.738401] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16339, [delay/us]:prepare:5, schedule:15, submit:14, rt:831, total:866) [2025-01-08 00:51:35.738422] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4195, iops:41, bw:171561, [delay/us]:prepare:6, schedule:121, submit:448, rt:1254, total:1831) [2025-01-08 00:51:35.738430] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:35.738456] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16339, [delay/us]: prepare:5, schedule:15, submit:14, device:831, total:866) [2025-01-08 00:51:35.738515] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4196, avg_iops:41, avg_bw:171555, [delay/us]: prepare:6, schedule:121, submit:448, device:1254, total:1831) [2025-01-08 00:51:35.738537] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:35.738559] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:580, total:590) [2025-01-08 00:51:35.738633] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:35.738671] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4084, [delay/us]: prepare:0, schedule:0, submit:600000000, device:580, total:590) [2025-01-08 00:51:35.738691] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:35.738707] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:35.738715] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:35.738723] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:35.738731] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=206kB/s, net_out=220kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:35.739399] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=115] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:35.739444] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=40] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:35.739460] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:35.739470] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:35.742155] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:51:35.743017] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=856] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:35.746276] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:35.757597] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:35.757678] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=63] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:35.757697] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=16] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:35.757705] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:35.757712] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:35.757721] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=101) [2025-01-08 00:51:35.759384] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:35.763282] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690F56-0-0] [lt=40] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.789499998092651367e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=894, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.777316481322638708e+01) [2025-01-08 00:51:35.764000] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3961, avg_group_log_size=396) [2025-01-08 00:51:35.771462] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=23] [APPLY STAT CB APPEND COST TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=158, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:35.771511] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=37] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=5659, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:35.771524] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:35.771531] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=-822, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:35.771538] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=39, stat_interval=5000000, avg (count/cost)=4995, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:35.774127] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.784441] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:35.786140] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=32] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:35.786355] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326294359805000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:35.786573] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=170] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:35.786644] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58563][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:35.787559] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58564][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:35.787675] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:51:35.787601] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58564][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C88-0-0] [lt=14] init point thread id with(&point=0x5572d50e5a40, idx_=1220, point=[thread id=58564, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58564) [2025-01-08 00:51:35.811096] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84380352}], scn_array=[cnt:1, {val:1736326295777461000, v:0}], curr_size=513, accum_size=2620, time_cost=13440) [2025-01-08 00:51:35.811301] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:35.811582] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:35.811670] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=87][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:35.811839] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:51:35.811972] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=61] ====== check clog disk timer task ====== [2025-01-08 00:51:35.811990] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=14] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:35.811991] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:35.812033] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=13] gc stale ls task succ [2025-01-08 00:51:35.812863] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=103] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1001}, {id:1002}]) [2025-01-08 00:51:35.813400] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:35.815821] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58552][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:35.818400] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:35.819697] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=42] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:35.819811] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=39] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326295661593000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:35.819838] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326295661593000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326295661593000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:35.824150] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D086-0-0] [lt=28][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:35.826744] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3EC-0-0] [lt=80][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:35.830995] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8710C-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:35.831039] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:35.831051] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:35.832275] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58563][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5C89-0-0] [lt=49] init point thread id with(&point=0x5572d50e5ac0, idx_=1222, point=[thread id=58563, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58563) [2025-01-08 00:51:35.833532] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:51:35.833711] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58565][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:35.835390] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326295835263000, start_id=1736326295835263000, end_id=1736326295835263001) [2025-01-08 00:51:35.849977] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918543A-0-0] [lt=85][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:35.850249] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:35.850276] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-01-08 00:51:35.850285] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:35.850298] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:35.850311] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193799758452, flush_io_finished_ret_:0, flush_io_finished_round_:1954, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:35.850348] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:35.850363] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-01-08 00:51:35.850368] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:35.850376] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:35.850384] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193799758527, flush_io_finished_ret_:0, flush_io_finished_round_:3782, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:35.850642] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=26] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:35.850973] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=55] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:35.851567] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2225542}], scn_array=[cnt:1, {val:1736326295328613004, v:0}], curr_size=113, accum_size=357, time_cost=19894) [2025-01-08 00:51:35.852461] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58566][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:35.854075] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:35.863906] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:35.865243] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=43] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:35.865244] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=19] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:35.865524] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=21] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326295766421000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:35.865601] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:51:35.865669] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=125] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:35.865727] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:35.865761] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43411059}], scn_array=[cnt:1, {val:1736326295847753000, v:0}], curr_size=589, accum_size=7272, time_cost=15148) [2025-01-08 00:51:35.865874] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:35.866140] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=43] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=4159488) [2025-01-08 00:51:35.868480] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:35.869484] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18705464}], scn_array=[cnt:1, {val:1736326295835263000, v:0}], curr_size=122, accum_size=1302, time_cost=21163) [2025-01-08 00:51:35.874370] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=110][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.877850] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get_max_applied_scn(scn={val:1736326295674610000, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2191906}, palf_committed_end_scn_:{val:1736326295835263000, v:0}, last_check_scn_:{val:1736326295835263000, v:0}, max_applied_cb_scn_:{val:1736326295674610000, v:0}}) [2025-01-08 00:51:35.877903] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=50] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2191906}, readable_log_tail:{lsn:2191906}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64912862, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:35.896441] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.896973] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.907256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=86] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.907301] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:35.916900] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1736326295916896, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007414, add_bytes: 0 [2025-01-08 00:51:35.917207] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39D9-0-0] [lt=34] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:35.917227] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39D9-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22202, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:35.917236] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39D9-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=4163, avg_log_submit_to_slide_time=22183) [2025-01-08 00:51:35.919471] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:35.926097] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58566][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4737-0-0] [lt=2391] init point thread id with(&point=0x5572d50e5b00, idx_=1223, point=[thread id=58566, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58566) [2025-01-08 00:51:35.935094] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:35.935178] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=76] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:35.935194] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=12] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:35.935275] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=78] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:35.938284] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=25] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:35.939991] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=38] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:35.941816] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=99][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2213256}], scn_array=[cnt:1, {val:1736326295819616000, v:0}], curr_size=122, accum_size=610, time_cost=24474) [2025-01-08 00:51:35.941943] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:35.946991] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739533824, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:35.950099] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:51:35.956225] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=43] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:35.956731] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=39] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:35.956781] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=48] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:35.958095] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487D-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:35.958180] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487D-0-0] [lt=39] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:35.958196] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487D-0-0] [lt=13] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:35.961098] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=79] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:35.963011] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=40] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:35.964603] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=39] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:35.964643] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:35.966088] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487D-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:35.966156] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487D-0-0] [lt=17] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:35.966168] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487D-0-0] [lt=11] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:35.976859] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:35.982772] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=11] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:35.984648] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=94] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:35.986176] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=42] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:35.986269] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=91] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:35.987594] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487D-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:35.987683] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487D-0-0] [lt=39] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:35.987700] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487D-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:35.987722] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=12] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:35.988351] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=625] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:35.988405] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=44] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:35.988416] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=8] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:35.988437] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487D-0-0] [lt=18] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:35.988457] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487D-0-0] [lt=19] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:35.988467] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=6] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:35.988473] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:35.988489] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487D-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:35.988501] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487D-0-0] [lt=12] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:35.988564] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=6] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:35.988576] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=12] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:35.988597] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487D-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:35.988611] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487D-0-0] [lt=14] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:35.989807] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:35.989861] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=50] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:35.989874] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=12] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:35.989909] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=8] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:35.989941] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=30] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:35.989953] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=11] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:35.989965] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:35.989980] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487D-0-0] [lt=14] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:35.990045] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=196, gc_interval=10000000) [2025-01-08 00:51:35.990082] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=16] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:35.990105] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=13] gc_check_member_list_ cost time(ret=0, time_us=42) [2025-01-08 00:51:35.990117] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:35.991331] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:35.991377] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=40] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:35.991392] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=14] execute_gc cost time(ret=0, time_us=17) [2025-01-08 00:51:35.991401] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyHandler start process [2025-01-08 00:51:36.006648] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDFE-0-0] [lt=32][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193800920483, cache_obj->added_lc()=false, cache_obj->get_object_id()=13654, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:36.008222] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598E-0-0] [lt=121] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12274, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:36.015012] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598E-0-0] [lt=159] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12274, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326296014940, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326296014940, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326296014940, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:36.015142] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598E-0-0] [lt=128] success to process renew task(task={tenant_id:1, table_id:12274, add_timestamp:1736326296007988}, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326296014940, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326296014940, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326296014940, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:36.027327] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:195) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CAE-0-0] [lt=47][errcode=-4018] table_operator_ get_source failed(ret=-4018) [2025-01-08 00:51:36.027572] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4739-0-0] [lt=11][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18286, log_task={header:{begin_lsn:{lsn:2225655}, end_lsn:{lsn:2225777}, log_id:18286, min_scn:{val:1736326295831186000, v:0}, max_scn:{val:1736326295831186000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2225542}, prev_proposal_id:1, committed_end_lsn:{lsn:2225655}, data_checksum:1715074492, accum_checksum:3769929195, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326295926171, freeze_ts:1736326295926171, submit_ts:1736326295926184, flushed_ts:1736326295932593, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:6409}, fs_cb_begin_ts=1736326296027559, log_life_time=101388) [2025-01-08 00:51:36.027715] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=71] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=611, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:36.027735] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:36.028958] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2213378}], scn_array=[cnt:1, {val:1736326295819616001, v:0}], curr_size=122, accum_size=732, time_cost=13467) [2025-01-08 00:51:36.032377] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B6-0-0] [lt=37] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:36.032419] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B6-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326295831186000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326295831186000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:36.033774] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CAE-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:36.035528] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CAE-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:36.039986] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=124, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:36.040019] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1912, avg_group_log_size=191) [2025-01-08 00:51:36.040037] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1912) [2025-01-08 00:51:36.040052] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:36.040123] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=37, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:36.040156] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:36.040172] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:36.040274] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=97] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:36.040411] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:300K/71:495871 disk:300K/71:495871 t1m2:48K/11:325390 t1m2g0:48K/11:325390 t1002m2:188K/46:311621 t1002m2g0:188K/46:311621 t1001m2:64K/14:189017 t1001m2g0:64K/14:189017 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:36.041785] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1912) [2025-01-08 00:51:36.041958] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=124] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:36.048151] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=81] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:36.053335] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=8450) [2025-01-08 00:51:36.053574] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A9C-0-0] [lt=67] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=14, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:36.053599] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A9C-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=5192, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:36.056156] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=93] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326295317734004, v:0}, new_max_version={val:1736326295317734004, v:0}, last_min_version={val:1736326295317734002, v:0}, last_max_version={val:1736326295317734002, v:0}, total_time_us=9513, delta_us=738419) [2025-01-08 00:51:36.056253] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=93] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326295317734004, v:0}, min_version={val:1736326295317734004, v:0}, max_version={val:1736326295317734004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:36.056766] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:36.060924] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58557][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=38] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:36.077104] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:36.088233] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5277442792, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:36.088273] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:36.088287] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8ac38bef0) [2025-01-08 00:51:36.088296] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5277442792, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:36.132957] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58564][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8F-0-0] [lt=159] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326296251702, keepalive_ts:-1}) [2025-01-08 00:51:36.132997] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58564][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD8F-0-0] [lt=39] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326296251702, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:36.136170] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=22] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:36.142108] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:36.144263] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB APPEND COST TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=83, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:36.144303] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=39] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=4724, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:36.144315] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:36.144322] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=-840, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:36.144329] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=3967, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:36.145449] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9428-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:36.145485] INFO [LIB] stat (utility.h:1210) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9428-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7517, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:36.150016] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:36.155143] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9789) [2025-01-08 00:51:36.177267] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:36.202619] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B4BBACD-0-0] [lt=52] get number(ret=0, service_type_=0, range=1, base_id=1736326296202309000, start_id=1736326296202309000, end_id=1736326296202309001) [2025-01-08 00:51:36.205609] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB APPEND COST TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=152, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:36.205637] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=28] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=6744, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:36.205646] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:36.205654] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=-731, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:36.205661] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=6164, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:36.213755] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39DF-0-0] [lt=46][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18186, log_task={header:{begin_lsn:{lsn:2213500}, end_lsn:{lsn:2213622}, log_id:18186, min_scn:{val:1736326295819616002, v:0}, max_scn:{val:1736326295819616002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2213378}, prev_proposal_id:1, committed_end_lsn:{lsn:2213500}, data_checksum:940009244, accum_checksum:3068244260, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326296113578, freeze_ts:1736326296113578, submit_ts:1736326296113591, flushed_ts:1736326296117496, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:3905}, fs_cb_begin_ts=1736326296213743, log_life_time=100165) [2025-01-08 00:51:36.226647] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58543][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A473D-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:36.241476] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2381, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:36.241513] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:36.241534] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1178, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:36.241694] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:36.242695] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87F6-0-0] [lt=64] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:36.242783] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87F6-0-0] [lt=87] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5557, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:36.242939] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:36.243945] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1003, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18187, max_lsn:2213744, max_scn:{val:1736326295819616003, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2213744}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2213622}, last_submit_end_lsn:{lsn:2213744}, last_submit_log_id:18187, last_submit_log_pid:1, max_flushed_lsn:{lsn:2213622}, max_flushed_end_lsn:{lsn:2213744}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2213744}, last_slide_log_id:18187, last_slide_scn:{val:1736326295819616003, v:0}, last_slide_lsn:{lsn:2213622}, last_slide_end_lsn:{lsn:2213744}, last_slide_log_pid:1, last_slide_log_accum_checksum:1745540543, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924fd6530}, [StateMgr]={this:0x2ad924fda830, self:"192.168.159.137:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.139:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18188, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad924fe12f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326294026734, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326294026734, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924fd7c70}, [ModeMgr]={palf_id:1003, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436825152001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/meta", dir_fd:181, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924fdca38, flashback_version:0}, log_storage:{log_tail:{lsn:2213744}, readable_log_tail:{lsn:2213744}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64891024, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, palf_epoch:4, last_purge_throttling_ts:-1, this:0x2ad924fdaf70}, [Reconfirm]={palf_id:1003, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924fda9b0}) [2025-01-08 00:51:36.244474] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=523][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:36.244491] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:36.244581] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:36.244606] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] current log_task status(palf_id=1003, self="192.168.159.137:2882", log_id=18188, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:51:36.249487] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=66, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:36.249530] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9789, avg_group_log_size=652) [2025-01-08 00:51:36.249549] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9789) [2025-01-08 00:51:36.250992] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=140] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=3209, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:36.251096] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=97] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=328, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:36.251153] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=25, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:36.253557] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3A9F-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=88, avg_log_submit_to_flush_time=3793, avg_log_submit_to_slide_time=5201) [2025-01-08 00:51:36.253931] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=109] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9789, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:36.254493] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58569][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:36.254749] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:51:36.258958] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A7D-0-0] [lt=15] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326296258920}) [2025-01-08 00:51:36.259088] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598F-0-0] [lt=27] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326296259067, sql_renew_time:1736326296259067, is_mark_fail:false}]) [2025-01-08 00:51:36.259124] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28598F-0-0] [lt=35] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326296258920}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326296259067, sql_renew_time:1736326296259067, is_mark_fail:false}]) [2025-01-08 00:51:36.267427] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326295966737000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=300681, in_cluster_service=true, cluster_version={val:1736326295777461000, v:0}, min_cluster_version={val:1736326295777461000, v:0}, max_cluster_version={val:1736326295777461000, v:0}, get_cluster_version_err=0, cluster_version_delta=489957, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36667, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=550764405, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:36.277742] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=99][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:36.278264] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3510, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:36.278314] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1921, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:36.278647] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:36.279405] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=28] ls blacklist refresh finish(cost_time=21580) [2025-01-08 00:51:36.283440] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:36.302180] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46495][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C695-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193801216010, cache_obj->added_lc()=false, cache_obj->get_object_id()=13655, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:36.313827] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39E1-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:36.313867] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39E1-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:36.313885] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39E1-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:36.314886] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5358, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:36.314931] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=289, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:36.315067] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:36.319054] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.319999] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=48] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.321992] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE8D5-0-0] [lt=36] get number(ret=0, service_type_=0, range=1, base_id=1736326296321952000, start_id=1736326296321952000, end_id=1736326296321952001) [2025-01-08 00:51:36.327070] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A473F-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:36.327119] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A473F-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:36.329545] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87F7-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:36.329572] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87F7-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31379, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:36.329582] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC87F7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=18, avg_log_submit_to_flush_time=3934, avg_log_submit_to_slide_time=31360) [2025-01-08 00:51:36.329894] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=344] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:36.329932] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18198, miss_cnt:0, cache_read_size:2226265, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:36.336713] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:36.336876] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=47][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:36.337135] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=40] ts response task statistics(alloc_count=60482, free_count=30240) [2025-01-08 00:51:36.337216] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=588, get_gts_with_stc_cnt=31, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=29, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:36.337235] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:36.337264] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=508, get_gts_with_stc_cnt=78, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=76, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:36.341798] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D16FBBE05-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1736326296339598000, start_id=1736326296339598000, end_id=1736326296339598001) [2025-01-08 00:51:36.342296] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:36.342358] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=26] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:36.344579] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=99] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:36.345739] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C7BB60E-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1736326296343612000, start_id=1736326296343612000, end_id=1736326296343612001) [2025-01-08 00:51:36.346669] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4177, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:36.346698] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2335) [2025-01-08 00:51:36.346730] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1800, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:36.346960] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=46, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:36.347650] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A942E-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=134, avg_log_submit_to_flush_time=6086, avg_log_submit_to_slide_time=7729) [2025-01-08 00:51:36.347825] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=81] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2335, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:36.350568] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=61] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:36.351109] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58571][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:36.351159] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:51:36.355392] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [50463][T1002_ArcFetche][T1002][YB42C0A89F89-00062B2D1C18458E-0-0] [lt=39] ObArchiveFetcher is running(thread_index=0) [2025-01-08 00:51:36.365841] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:36.365919] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:36.366000] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:36.366340] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:36.366558] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=126] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:36.370443] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:36.377886] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:36.402597] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC6-0-0] [lt=14] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:36.402634] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC6-0-0] [lt=33] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:36.414098] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1736326296414095, bytes: 879130753, bw: 0.258985 MB/s, add_ts: 1002463, add_bytes: 272234 [2025-01-08 00:51:36.415563] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:36.427942] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3654, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:36.451977] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E0A-0-0] [lt=189] init point thread id with(&point=0x5572d50e5c40, idx_=1228, point=[thread id=58571, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58571) [2025-01-08 00:51:36.459037] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=27] gc stale ls task succ [2025-01-08 00:51:36.459386] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.459437] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.474420] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5289) [2025-01-08 00:51:36.474452] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:36.475493] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5289) [2025-01-08 00:51:36.475636] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=5289) [2025-01-08 00:51:36.477532] INFO [LIB] stat (utility.h:1210) [58564][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6928-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:36.477567] INFO [LIB] stat (utility.h:1210) [58564][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6928-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=6513, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:36.477581] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58564][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6928-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=65, avg_log_gen_to_submit_time=179, avg_log_submit_to_flush_time=5478, avg_log_submit_to_slide_time=6334) [2025-01-08 00:51:36.477688] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=115] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5289, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:36.478019] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:36.519609] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.527777] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=55] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:36.527836] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=51] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=218, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:36.528007] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=114] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:36.542516] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:36.542730] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.544384] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=64] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.551247] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=44] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:36.569909] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=87, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:36.578221] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:36.669073] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60D-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:36.669119] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60D-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326296320812002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326296320812002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:36.674630] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1A1-0-0] [lt=15] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:36.674665] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1A1-0-0] [lt=28] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:36.674694] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1A1-0-0] [lt=27] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:36.678387] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:36.678421] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:36.701839] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=139] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:36.717386] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18102, miss_cnt:0, cache_read_size:2214354, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:36.733009] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=26] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8740, offset=2481850}, end_cursor:ObLogCursor{file_id=1, log_id=8741, offset=2482155}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b84050, len:4422, log_data_len:305, seq:8740, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:36.733625] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E64-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:36.733679] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E64-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326296333349003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326296333349003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:36.737518] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E64-0-0] [lt=44] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326296737422, can_tenant_elr:true}) [2025-01-08 00:51:36.741104] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:36.741155] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:53084, [delay/us]:prepare:10, schedule:305, submit:275, rt:1728, total:2320) [2025-01-08 00:51:36.741171] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:36.741306] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=119] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:53080, [delay/us]: prepare:10, schedule:305, submit:275, device:1728, total:2320) [2025-01-08 00:51:36.741321] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:36.741472] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:14, bw:65334, [delay/us]:prepare:7, schedule:201, submit:226, rt:1604, total:2040) [2025-01-08 00:51:36.741487] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:36.741587] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:14, avg_bw:65329, [delay/us]: prepare:7, schedule:201, submit:226, device:1604, total:2040) [2025-01-08 00:51:36.741740] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=145] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:36.741767] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:8192, iops:1, bw:8164, [delay/us]:prepare:6, schedule:16, submit:1, rt:1073, total:1096) [2025-01-08 00:51:36.741816] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4201, iops:39, bw:163290, [delay/us]:prepare:7, schedule:788, submit:477, rt:2042, total:3315) [2025-01-08 00:51:36.741888] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=67] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:36.741921] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:8192, avg_iops:1, avg_bw:8163, [delay/us]: prepare:6, schedule:16, submit:1, device:1073, total:1096) [2025-01-08 00:51:36.741938] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4201, avg_iops:39, avg_bw:163280, [delay/us]: prepare:7, schedule:788, submit:477, device:2042, total:3315) [2025-01-08 00:51:36.741948] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:36.741968] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4082, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:4062, total:4083) [2025-01-08 00:51:36.741992] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:36.742018] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4082, [delay/us]: prepare:0, schedule:0, submit:600000000, device:4062, total:4083) [2025-01-08 00:51:36.742036] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:36.742051] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:36.742059] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:36.742067] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:36.742074] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=251kB/s, net_out=289kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:36.743694] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=31] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1384064, replace_num=15728) [2025-01-08 00:51:36.743736] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:36.745802] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8741, offset=2482155}, end_cursor:ObLogCursor{file_id=1, log_id=8742, offset=2482452}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b88050, len:4117, log_data_len:297, seq:8741, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:36.746474] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E64-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:36.746516] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E64-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326296333349003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326296333349003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:36.751051] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085165-0-0] [lt=21] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:36.752207] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=56] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:36.757238] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8742, offset=2482452}, end_cursor:ObLogCursor{file_id=1, log_id=8743, offset=2482749}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b8c050, len:3820, log_data_len:297, seq:8742, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:36.758224] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8743, offset=2482749}, end_cursor:ObLogCursor{file_id=1, log_id=8744, offset=2483046}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b94050, len:3523, log_data_len:297, seq:8743, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:36.769773] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690F65-0-0] [lt=28] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.787099943161010742e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=893, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.792224504070524205e+01) [2025-01-08 00:51:36.770165] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=5290, avg_group_log_size=440) [2025-01-08 00:51:36.774635] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=162] ====== tenant freeze timer task ====== [2025-01-08 00:51:36.774703] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:689) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=43] TxData Memory Statistic(logstream info): (ls_id={id:1}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=11) [2025-01-08 00:51:36.774722] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:595) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=15] TxData Memory Statistic : (Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=11, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=14, Total TxDataTable Memory Limit(MB)=409) [2025-01-08 00:51:36.778560] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:36.805565] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3EF-0-0] [lt=40][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:36.811486] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D089-0-0] [lt=54][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:36.811865] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:36.812049] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:36.812070] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:36.812368] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=13] gc stale ls task succ [2025-01-08 00:51:36.812409] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:36.812447] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=9] gc stale ls task succ [2025-01-08 00:51:36.813134] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1001}, {id:1002}]) [2025-01-08 00:51:36.817099] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87112-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:36.817138] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:36.817150] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:36.826004] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:36.833779] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F56E-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1736326296833575000, start_id=1736326296833575000, end_id=1736326296833575001) [2025-01-08 00:51:36.834767] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:36.835166] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58564][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:51:36.835489] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:36.835510] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=59] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:36.835523] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=181) [2025-01-08 00:51:36.835539] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=22] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:36.835552] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=12] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=192) [2025-01-08 00:51:36.837916] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326296837882], gts:1736326296837226000, latest_srr:[mts=1736326296837882]}) [2025-01-08 00:51:36.837961] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326296837950], gts:1736326296837226000, latest_srr:[mts=1736326296837950]}) [2025-01-08 00:51:36.838298] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326296838175], gts:1736326296837226000, latest_srr:[mts=1736326296838175]}) [2025-01-08 00:51:36.845302] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:36.845513] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58543][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=51] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:36.851403] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=80] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:36.852508] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918543B-0-0] [lt=85][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:36.866093] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:36.866151] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:36.866270] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:36.866507] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:36.866755] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:36.870589] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:36.876401] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58565][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA692C-0-0] [lt=20] init point thread id with(&point=0x5572d50e5b40, idx_=1224, point=[thread id=58565, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58565) [2025-01-08 00:51:36.880377] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:36.897249] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.897249] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.897413] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.897684] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:36.915015] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39ED-0-0] [lt=18][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18193, log_task={header:{begin_lsn:{lsn:2214354}, end_lsn:{lsn:2214476}, log_id:18193, min_scn:{val:1736326296320812004, v:0}, max_scn:{val:1736326296320812004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2214232}, prev_proposal_id:1, committed_end_lsn:{lsn:2214354}, data_checksum:940009244, accum_checksum:3668691027, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326296814793, freeze_ts:1736326296814794, submit_ts:1736326296814807, flushed_ts:1736326296818430, gen_to_freeze cost time:1, gen_to_submit cost time:14, submit_to_flush cost time:3623}, fs_cb_begin_ts=1736326296914989, log_life_time=100196) [2025-01-08 00:51:36.919397] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A943E-0-0] [lt=102] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:36.919466] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A943E-0-0] [lt=68] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=30297, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:36.919478] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A943E-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=23, avg_log_submit_to_flush_time=5271, avg_log_submit_to_slide_time=30274) [2025-01-08 00:51:36.919545] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:36.923383] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1736326296923378, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006482, add_bytes: 0 [2025-01-08 00:51:36.930448] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF30-0-0] [lt=37] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326296950067, keepalive_ts:-1}) [2025-01-08 00:51:36.930506] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF30-0-0] [lt=56] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326296950067, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:36.935321] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58572][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:36.935432] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8809-0-0] [lt=26] init point thread id with(&point=0x5572d50e5cc0, idx_=1230, point=[thread id=58572, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58572) [2025-01-08 00:51:36.936027] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C4-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326296947185, keepalive_ts:-1}) [2025-01-08 00:51:36.936055] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C4-0-0] [lt=27] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326296947185, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:36.936634] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C5-0-0] [lt=24] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326296947370, keepalive_ts:-1}) [2025-01-08 00:51:36.936716] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C5-0-0] [lt=80] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326296947370, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:36.937933] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:36.944159] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE3-0-0] [lt=255] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:36.944180] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:36.944256] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE3-0-0] [lt=87] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=99) [2025-01-08 00:51:36.948944] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get_min_unreplayed_log_info(lsn={lsn:2226875}, scn={val:1736326296332213005, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326296829837, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2226875}, next_to_submit_scn_:{val:1736326296332213005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2226753}, end_lsn:{lsn:2226875}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2226997}, readable_log_tail:{lsn:2226997}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64877771, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326296332213004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326296332213004, v:0}, data_checksum:3348837161, flag:62768}}, init_mode_version:0, accumulate_checksum:2940436555, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326296332213004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18203, miss_cnt:0, cache_read_size:2226875, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:36.952738] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:36.954560] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=57] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1069538, clean_num=31457) [2025-01-08 00:51:36.960037] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:36.960514] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58573][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:36.960589] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1001, group_id_=1, token=8) [2025-01-08 00:51:36.968577] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [46323][T1_RFLWorker][T1][YB42C0A89F89-00062B2D1618458E-0-0] [lt=23] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:51:36.980549] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.028256] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=138] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=80, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:37.028290] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=34] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:37.043693] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=70, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:37.043744] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:37.043798] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=39, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:37.043832] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:37.045296] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CAF-0-0] [lt=53][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:37.046539] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326293024370, last_first_register_time_us=1736324449183670, register_time_us=1736326293024370, curr_time_us=1736326297046536) [2025-01-08 00:51:37.046623] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326297046573, last_submit_register_req_time_us=1736326297046573, register_parent_reason=1) [2025-01-08 00:51:37.046641] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:37.047614] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC880C-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:37.047737] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=16] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:312K/70:189833 disk:312K/70:189833 t1m2:52K/11:426990 t1m2g0:52K/11:426990 t1002m2:188K/44:138626 t1002m2g0:188K/44:138626 t1001m2:72K/15:166127 t1001m2g0:72K/15:166127 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:37.048246] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58565][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F9-0-0] [lt=122] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326297055934, keepalive_ts:-1}) [2025-01-08 00:51:37.048276] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58565][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0F9-0-0] [lt=30] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326297055934, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:37.049576] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CAF-0-0] [lt=85][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:37.049951] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=153] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326296320812004, v:0}, base_version={val:1736326296320812002, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326296332213004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326297049851, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326296332213004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326297151639, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326296320812004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326297054795, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:37.051969] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:37.052674] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9442-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:51:37.056132] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9818) [2025-01-08 00:51:37.056386] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58573][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E13-0-0] [lt=47] init point thread id with(&point=0x5572d50e5d00, idx_=1231, point=[thread id=58573, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58573) [2025-01-08 00:51:37.056545] INFO [LIB] stat (utility.h:1210) [58573][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E13-0-0] [lt=85] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=23, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:37.056609] INFO [LIB] stat (utility.h:1210) [58573][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E13-0-0] [lt=63] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=3498, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:37.059395] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326296320812004, v:0}, new_max_version={val:1736326296320812004, v:0}, last_min_version={val:1736326296320812002, v:0}, last_max_version={val:1736326296320812002, v:0}, total_time_us=9401, delta_us=738581) [2025-01-08 00:51:37.059548] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=43] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326296320812004, v:0}, min_version={val:1736326296320812004, v:0}, max_version={val:1736326296320812004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:37.059647] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=77] get_max_applied_scn(scn={val:1736326296926294000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18707896}, palf_committed_end_scn_:{val:1736326296926294001, v:0}, last_check_scn_:{val:1736326296926294001, v:0}, max_applied_cb_scn_:{val:1736326296926294000, v:0}}) [2025-01-08 00:51:37.059675] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18707896}, readable_log_tail:{lsn:18707896}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48396872, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:37.059796] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=99] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:37.076762] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=97] LogLoopThread round_cost_time(us)(round_cost_time=13) [2025-01-08 00:51:37.081087] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=92] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:37.081123] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=100) [2025-01-08 00:51:37.081575] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.081980] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B7-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:37.082021] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B7-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326296831532001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326296831532001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:37.088970] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5279124039, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:37.089008] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:37.089020] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad908547e10) [2025-01-08 00:51:37.089029] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5279124039, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:37.101524] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50612][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C2B7-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326297100619000, start_id=1736326297100619000, end_id=1736326297100619001) [2025-01-08 00:51:37.102292] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=8] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84386081}, readable_log_tail:{lsn:84386081}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49823455, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:37.102392] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=97] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:37.102425] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=12] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:37.119074] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=197, gc_interval=10000000) [2025-01-08 00:51:37.119509] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=28] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:37.119530] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:37.123047] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:51:37.123998] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=22] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:51:37.124025] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=23] gc_check_member_list_ cost time(ret=0, time_us=4923) [2025-01-08 00:51:37.124041] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:37.125742] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:37.127260] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=20] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:37.128841] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:37.130660] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:37.130685] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:37.130701] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=16] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:37.130713] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:37.130724] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:37.130734] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=51) [2025-01-08 00:51:37.130743] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:51:37.132747] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58566][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF60-0-0] [lt=56] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326297250001, keepalive_ts:-1}) [2025-01-08 00:51:37.132781] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58566][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF60-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326297250001, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:37.144438] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:37.144978] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=2188, avg_group_log_size=198) [2025-01-08 00:51:37.145022] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2188) [2025-01-08 00:51:37.145108] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:37.145124] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:37.152622] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=30] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:37.154165] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:37.154261] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5981, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:37.158158] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:37.158373] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2193248}], scn_array=[cnt:1, {val:1736326296926294001, v:0}], curr_size=122, accum_size=976, time_cost=13086) [2025-01-08 00:51:37.159348] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=145] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:37.159430] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9775) [2025-01-08 00:51:37.181661] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.192498] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=32] ====== check clog disk timer task ====== [2025-01-08 00:51:37.192543] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=38] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:37.235518] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58566][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4751-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:37.239353] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15] ====== traversal_flush timer task ====== [2025-01-08 00:51:37.239469] INFO [MDS] flush (mds_table_mgr.cpp:195) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48539D-0-0] [lt=27] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326296926294002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326294324153003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326310087570000, latest_log_ts:{val:1736326293821063001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.239634] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48539D-0-0] [lt=101] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326292820856000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326294324153003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.239727] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48539D-0-0] [lt=63] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326292324052004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326294324153003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.239840] INFO [MDS] flush (mds_table_mgr.cpp:172) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48539D-0-0] [lt=80] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326292316531003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326294324153003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.239901] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D48539D-0-0] [lt=58] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:37.248676] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=4006, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:37.248718] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:37.248738] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=411, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:37.248805] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=26, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:37.250532] INFO [LIB] stat (utility.h:1210) [58402][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8813-0-0] [lt=47] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:37.250572] INFO [LIB] stat (utility.h:1210) [58402][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8813-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=5626, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:37.250746] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=86] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:37.255102] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=63, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:37.255141] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=10364, avg_group_log_size=647) [2025-01-08 00:51:37.255161] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10364) [2025-01-08 00:51:37.258300] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2335, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:37.258397] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=258, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:37.258689] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=49] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=43, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:37.259619] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58573][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E17-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=135, avg_log_gen_to_submit_time=212, avg_log_submit_to_flush_time=2729, avg_log_submit_to_slide_time=3660) [2025-01-08 00:51:37.259760] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10364, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:37.280462] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=66] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2338, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:37.280504] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=503, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:37.280568] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=197] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=50, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:37.282270] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:37.282304] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.283757] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=39] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:37.284526] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=9] ====== check clog disk timer task ====== [2025-01-08 00:51:37.284568] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=33] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:37.318987] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.319043] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.331905] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C697-0-0] [lt=78][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193802245739, cache_obj->added_lc()=false, cache_obj->get_object_id()=13656, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:37.335645] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=82] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:37.335735] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=82] temporary file module use no blocks [2025-01-08 00:51:37.335747] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:37.335761] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:37.335775] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193801249213, flush_io_finished_ret_:0, flush_io_finished_round_:1960, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:37.338748] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:37.339930] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4753-0-0] [lt=25] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:37.339968] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4753-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:37.341545] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2231459, this={dio_aligned_buf:{buf_write_offset:3357, buf_padding_size:739, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:26671, truncate_used_ts:5482}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2228224, buf_len=122, total_write_size=12172, total_write_size_after_dio=421888, ob_pwrite_used_ts=225703, count=100, each_pwrite_cost=2257) [2025-01-08 00:51:37.344726] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=84] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:37.344940] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D962-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=37, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:37.344963] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D962-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12848, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:37.344973] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D962-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=1620, avg_log_submit_to_slide_time=12826) [2025-01-08 00:51:37.345063] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:37.345289] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=215] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18208, miss_cnt:0, cache_read_size:2227485, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:37.348750] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8816-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=101, avg_log_submit_to_flush_time=4169, avg_log_submit_to_slide_time=5713) [2025-01-08 00:51:37.348863] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1643, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:37.362804] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=94] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:37.364440] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D48-0-0] [lt=36] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:37.366347] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:37.366478] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:37.366870] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:37.367061] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=73] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:37.367319] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:37.371626] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=50] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:37.380456] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=27] current monitor number(seq_=-1) [2025-01-08 00:51:37.382355] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.404390] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC7-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:37.404425] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC7-0-0] [lt=32] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:37.416166] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1736326297416163, bytes: 879457673, bw: 0.311132 MB/s, add_ts: 1002068, add_bytes: 326920 [2025-01-08 00:51:37.416345] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39F5-0-0] [lt=24] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:37.416503] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39F5-0-0] [lt=134] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:37.416525] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39F5-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:37.417662] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1722, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:37.417703] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:37.417788] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=299, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:37.417878] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=48, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:37.427532] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=159] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:37.427703] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] ====== traversal_flush timer task ====== [2025-01-08 00:51:37.427875] INFO [MDS] flush (mds_table_mgr.cpp:195) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F28-0-0] [lt=43] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326297348621000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309284252000, latest_log_ts:{val:1736326292617740001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.428059] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F28-0-0] [lt=177] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:37.435495] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1293, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:37.447353] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3342, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:37.447384] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1765) [2025-01-08 00:51:37.447504] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=111] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=533, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:37.447569] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=32, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:37.457683] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=35] ====== tenant freeze timer task ====== [2025-01-08 00:51:37.457764] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:689) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=45] TxData Memory Statistic(logstream info): (ls_id={id:1}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=8) [2025-01-08 00:51:37.457781] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:689) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15] TxData Memory Statistic(logstream info): (ls_id={id:1001}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=16) [2025-01-08 00:51:37.457796] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:689) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14] TxData Memory Statistic(logstream info): (ls_id={id:1002}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=16) [2025-01-08 00:51:37.457804] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:689) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6] TxData Memory Statistic(logstream info): (ls_id={id:1003}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=16) [2025-01-08 00:51:37.457813] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:595) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=5] TxData Memory Statistic : (Tenant Total Memory(MB)=3072, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=56, Freeze TxData Trigger Memory(MB)=61, Total TxDataTable Hold Memory(MB)=66, Total TxDataTable Memory Limit(MB)=614) [2025-01-08 00:51:37.459182] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=45] gc stale ls task succ [2025-01-08 00:51:37.459413] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.459451] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.465485] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=36] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:37.465530] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=40] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:37.465544] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:37.465594] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:37.468545] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] Dumping All Cache Objs(alloc_obj_list.count()=132, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}]) [2025-01-08 00:51:37.468972] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=423] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:37.470130] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] ====== tenant freeze timer task ====== [2025-01-08 00:51:37.470171] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=35] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71124160, tenant_id=1001) [2025-01-08 00:51:37.470258] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=77] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:37.477959] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3963) [2025-01-08 00:51:37.477987] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:37.480096] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3963) [2025-01-08 00:51:37.480361] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3963) [2025-01-08 00:51:37.480528] INFO [LIB] stat (utility.h:1210) [58565][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6932-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:37.480546] INFO [LIB] stat (utility.h:1210) [58565][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6932-0-0] [lt=18] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4428, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:37.480556] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58565][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6932-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=120, avg_log_submit_to_flush_time=2859, avg_log_submit_to_slide_time=4308) [2025-01-08 00:51:37.480758] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3963, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:37.482485] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.518453] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=66] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:37.519554] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.534706] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:37.534741] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=274, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:37.534808] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=45, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:37.541571] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=91] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.544279] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=155] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.544946] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:37.563403] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=87] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:37.564595] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58569][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3AB2-0-0] [lt=461] init point thread id with(&point=0x5572d50e5bc0, idx_=1226, point=[thread id=58569, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58569) [2025-01-08 00:51:37.571951] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=78] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=101, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:37.582636] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.645793] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=64] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:37.645891] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=95] try gc retain ctx [2025-01-08 00:51:37.645902] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-01-08 00:51:37.645971] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326297338357002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326287305763002, v:0}, keep_alive_scn_:{val:1736326292316531004, v:0}, update_ts_:1736326297645833}) [2025-01-08 00:51:37.646754] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=81] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326294324153003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326310087570000, latest_log_ts:{val:1736326293821063001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:37.647057] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=149] finish update min start scn(max_decided_scn={val:1736326297338357002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326287305763002, v:0}, keep_alive_scn_:{val:1736326292316531004, v:0}, update_ts_:1736326297645833}) [2025-01-08 00:51:37.647906] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326292820856000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326294324153003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:37.648108] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=124] finish update min start scn(max_decided_scn={val:1736326297339561001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326290323279004, v:0}, keep_alive_scn_:{val:1736326295328613004, v:0}, update_ts_:1736326297647152}) [2025-01-08 00:51:37.648926] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=35] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326292324052004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326294324153003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:37.649022] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=88] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:37.649045] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:2215330}, scn={val:1736326297322585003, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326297618997, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2215330}, next_to_submit_scn_:{val:1736326297322585003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2215208}, end_lsn:{lsn:2215330}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2215330}, readable_log_tail:{lsn:2215330}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64889438, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326297322585002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326297322585002, v:0}, data_checksum:2124447694, flag:26861}}, init_mode_version:0, accumulate_checksum:4250931921, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326297322585002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18110, miss_cnt:0, cache_read_size:2215330, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:37.649117] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=71] finish update min start scn(max_decided_scn={val:1736326297322585002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326289313850001, v:0}, keep_alive_scn_:{val:1736326294317339001, v:0}, update_ts_:1736326297648509}) [2025-01-08 00:51:37.649867] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326292316531003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326294324153003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:37.682779] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.711837] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60E-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:37.711878] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60E-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326297322585002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326297322585002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:37.720744] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:37.720769] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:37.720864] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498C-0-0] [lt=13] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:37.720914] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498C-0-0] [lt=47] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:37.725185] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB60E-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326297724680000, start_id=1736326297724680000, end_id=1736326297724680001) [2025-01-08 00:51:37.725478] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:149) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB60E-0-0] [lt=97] transaction rpc statistics(total_rt=5347, total_process=18, avg_rt=281) [2025-01-08 00:51:37.728083] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=30] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:37.745057] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=105] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:37.745929] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:37.746005] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44838, [delay/us]:prepare:13, schedule:249, submit:97, rt:1581, total:1941) [2025-01-08 00:51:37.746094] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:37.746282] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=92] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44833, [delay/us]: prepare:13, schedule:249, submit:97, device:1581, total:1941) [2025-01-08 00:51:37.746302] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:37.746346] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:14, bw:65210, [delay/us]:prepare:7, schedule:270, submit:349, rt:1445, total:2071) [2025-01-08 00:51:37.746358] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:37.746401] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:14, avg_bw:65218, [delay/us]: prepare:7, schedule:270, submit:349, device:1445, total:2071) [2025-01-08 00:51:37.746411] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:37.746470] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5461, iops:3, bw:16307, [delay/us]:prepare:4, schedule:29, submit:0, rt:670, total:704) [2025-01-08 00:51:37.746496] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:38, bw:154920, [delay/us]:prepare:6, schedule:302, submit:326, rt:1694, total:2330) [2025-01-08 00:51:37.746506] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:37.746537] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5461, avg_iops:3, avg_bw:16308, [delay/us]: prepare:4, schedule:29, submit:0, device:670, total:704) [2025-01-08 00:51:37.746553] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:38, avg_bw:154932, [delay/us]: prepare:6, schedule:302, submit:326, device:1694, total:2330) [2025-01-08 00:51:37.746563] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:37.746583] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4077, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1547, total:1555) [2025-01-08 00:51:37.746607] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:37.746633] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4077, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1547, total:1555) [2025-01-08 00:51:37.746649] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:37.746687] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:37.746697] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:37.746705] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:37.746712] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=311kB/s, net_out=229kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:37.763131] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=51] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8744, offset=2483046}, end_cursor:ObLogCursor{file_id=1, log_id=8745, offset=2483351}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b90050, len:3226, log_data_len:305, seq:8744, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:37.764798] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=103] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:37.766049] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E65-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:37.766174] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E65-0-0] [lt=110] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326297338357004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326297338357004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:37.772334] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326297768759000, start_id=1736326297768759000, end_id=1736326297768759001) [2025-01-08 00:51:37.780324] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=37] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8745, offset=2483351}, end_cursor:ObLogCursor{file_id=1, log_id=8746, offset=2483648}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b98050, len:2921, log_data_len:297, seq:8745, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:37.780994] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E65-0-0] [lt=82] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:37.781054] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E65-0-0] [lt=44] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326297338357004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326297338357004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:37.782928] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.790712] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=46] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8746, offset=2483648}, end_cursor:ObLogCursor{file_id=1, log_id=8747, offset=2483945}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894b9c050, len:2624, log_data_len:297, seq:8746, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:37.791845] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8747, offset=2483945}, end_cursor:ObLogCursor{file_id=1, log_id=8748, offset=2484242}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ba0050, len:2327, log_data_len:297, seq:8747, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:37.792134] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18110, miss_cnt:0, cache_read_size:2215330, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:37.808081] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3F2-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:37.811635] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:37.811657] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:37.812086] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=76] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:37.812305] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] ====== check clog disk timer task ====== [2025-01-08 00:51:37.812409] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:37.812766] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:51:37.812855] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:37.812902] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=8] gc stale ls task succ [2025-01-08 00:51:37.815062] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D08C-0-0] [lt=81][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:37.815470] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=31] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1003}, {id:1}, {id:1002}]) [2025-01-08 00:51:37.817767] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87118-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:37.817842] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:37.817856] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:37.818288] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC39FD-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18201, log_task={header:{begin_lsn:{lsn:2215330}, end_lsn:{lsn:2215452}, log_id:18201, min_scn:{val:1736326297322585003, v:0}, max_scn:{val:1736326297322585003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2215208}, prev_proposal_id:1, committed_end_lsn:{lsn:2215330}, data_checksum:940009244, accum_checksum:2656169627, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326297716328, freeze_ts:1736326297716328, submit_ts:1736326297716424, flushed_ts:1736326297719527, gen_to_freeze cost time:0, gen_to_submit cost time:96, submit_to_flush cost time:3103}, fs_cb_begin_ts=1736326297818277, log_life_time=101949) [2025-01-08 00:51:37.828481] INFO pn_ratelimit (group.c:653) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:37.837681] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=19] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:37.837719] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=28] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=94) [2025-01-08 00:51:37.840805] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=69] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=647712768, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=426029056, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:37.840886] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:37.840907] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2859, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:37.840967] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:37.841017] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:37.841030] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:37.843352] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:37.843933] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58559][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=13] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:51:37.851643] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=23] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:37.854953] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918543C-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:37.855179] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:37.855322] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=134] temporary file module use no blocks [2025-01-08 00:51:37.855334] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:37.855348] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:37.855362] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193801766614, flush_io_finished_ret_:0, flush_io_finished_round_:1956, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:37.855410] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=80] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:37.855427] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-01-08 00:51:37.855433] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:37.855442] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:37.855452] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193801766623, flush_io_finished_ret_:0, flush_io_finished_round_:3784, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:37.861927] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AE40-0-0] [lt=12] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.840757417678833008e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=392, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.429380301347038085e+02) [2025-01-08 00:51:37.866564] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=46] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:37.867109] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:37.867161] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:37.867334] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:37.867757] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:37.867840] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:37.878070] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857B9-0-0] [lt=86][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193802791899, cache_obj->added_lc()=false, cache_obj->get_object_id()=13657, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:37.878393] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:37.880078] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] ====== traversal_flush timer task ====== [2025-01-08 00:51:37.880348] INFO [MDS] flush (mds_table_mgr.cpp:195) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857D5-0-0] [lt=37] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326297768759000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309254996000, latest_log_ts:{val:1736326292647912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.880523] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857D5-0-0] [lt=146] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:37.881048] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857BA-0-0] [lt=54][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:37.882913] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857BC-0-0] [lt=23][errcode=-4201] hashtable not init [2025-01-08 00:51:37.884947] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=89][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:37.896845] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.897299] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.897336] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.897381] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:37.913541] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=85] ====== checkpoint timer task ====== [2025-01-08 00:51:37.914394] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=33] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:37.914437] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=44] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:37.914507] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326295325887003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326310087570000, latest_log_ts:{val:1736326293821063001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.914610] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=98] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:37.914630] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:37.914644] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1736326293821063001, v:0}) [2025-01-08 00:51:37.914652] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:37.914667] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:37.914685] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:37.914697] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:37.914716] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:37.914738] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:37.915466] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:37.915546] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=79] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:37.915586] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326292820856000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326295325887003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.915643] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=56] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:37.915655] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:37.915667] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:37.915680] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:37.915696] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2194215}, scn={val:1736326297838903000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:37.923095] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8748, offset=2484242}, end_cursor:ObLogCursor{file_id=1, log_id=8749, offset=2484539}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ba4050, len:2030, log_data_len:297, seq:8748, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:37.923343] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326297838903000, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:37.923384] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:37.923407] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:37.924123] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:37.924160] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=37] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:37.924301] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=90] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326292324052004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326295325887003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.924375] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=72] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:37.924390] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:37.924408] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:37.924424] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:37.924438] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2227973}, scn={val:1736326297339561003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:37.925008] INFO [PALF] pread (log_reader.cpp:130) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=399) [2025-01-08 00:51:37.925961] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D96D-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=32, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:37.925989] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D96D-0-0] [lt=28] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=25592, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:37.925999] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58572][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D96D-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=42, avg_log_submit_to_flush_time=2453, avg_log_submit_to_slide_time=25550) [2025-01-08 00:51:37.926151] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:37.926702] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=112] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8749, offset=2484539}, end_cursor:ObLogCursor{file_id=1, log_id=8750, offset=2484836}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ba8050, len:1733, log_data_len:297, seq:8749, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:37.926825] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326297339561003, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:37.926859] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:37.926879] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:37.927698] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:37.927732] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=34] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:37.927773] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326292316531003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326295325887003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:37.927835] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=59] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:37.927848] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:37.927861] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:37.927875] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:37.927887] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2215696}, scn={val:1736326297823944000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:37.928563] INFO [PALF] pread (log_reader.cpp:130) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=5128) [2025-01-08 00:51:37.929836] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=48] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8750, offset=2484836}, end_cursor:ObLogCursor{file_id=1, log_id=8751, offset=2485133}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bb0050, len:1436, log_data_len:297, seq:8750, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:37.929958] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326297823944000, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:37.929990] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:37.930010] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:37.930020] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:37.931013] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=61] PNIO [ratelimit] time: 1736326297931010, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007632, add_bytes: 0 [2025-01-08 00:51:37.943427] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:37.943874] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58566][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=160] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:37.945313] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=81] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:37.966518] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:37.966740] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:37.968694] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58573][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=22] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:51:37.974888] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:37.979962] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4085, avg_group_log_size=371) [2025-01-08 00:51:37.985265] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.034940] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=51, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:38.034971] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:38.047625] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=124] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=80, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:38.047694] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:38.047833] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:38.047869] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:38.048352] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/74:219017 disk:308K/74:219017 t1m2:40K/10:54524 t1m2g0:40K/10:54524 t1002m2:200K/49:201603 t1002m2g0:200K/49:201603 t1001m2:68K/15:385567 t1001m2g0:68K/15:385567 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:38.052262] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC882A-0-0] [lt=56] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:38.059601] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB0-0-0] [lt=51][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:38.060268] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326294026734, last_first_register_time_us=1736324454612308, register_time_us=1736326294026734, curr_time_us=1736326298060265) [2025-01-08 00:51:38.060356] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=51] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326298060321, last_submit_register_req_time_us=1736326298060321, register_parent_reason=1) [2025-01-08 00:51:38.060372] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:38.060612] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BAF-0-0] [lt=47] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:38.060806] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BB1-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:38.062659] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=61] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:38.063225] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2194337}], scn_array=[cnt:1, {val:1736326297838903002, v:0}], curr_size=122, accum_size=967, time_cost=15128) [2025-01-08 00:51:38.063940] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18708985}], scn_array=[cnt:1, {val:1736326297838903002, v:0}], curr_size=122, accum_size=723, time_cost=16202) [2025-01-08 00:51:38.064473] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:51:38.066736] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB0-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:38.067861] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9818) [2025-01-08 00:51:38.072507] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E22-0-0] [lt=55] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:38.072533] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E22-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=4536, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:38.075077] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C699-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193802988910, cache_obj->added_lc()=false, cache_obj->get_object_id()=13658, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:38.075954] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326297322585004, v:0}, new_max_version={val:1736326297322585004, v:0}, last_min_version={val:1736326297322585002, v:0}, last_max_version={val:1736326297322585002, v:0}, total_time_us=15051, delta_us=753366) [2025-01-08 00:51:38.075991] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326297322585004, v:0}, min_version={val:1736326297322585004, v:0}, max_version={val:1736326297322585004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:38.076017] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:38.085546] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.089955] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C69E-0-0] [lt=79] get number(ret=0, service_type_=0, range=1, base_id=1736326298089493000, start_id=1736326298089493000, end_id=1736326298089493001) [2025-01-08 00:51:38.090646] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5280953105, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:38.090682] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:38.090694] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a1eb0) [2025-01-08 00:51:38.090703] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5280953105, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:38.097263] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:38.098296] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58576][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:38.098464] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:51:38.129597] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B8-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:38.129701] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B8-0-0] [lt=48] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326297835365001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326297835365001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:38.137432] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF61-0-0] [lt=29] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326298254297, keepalive_ts:-1}) [2025-01-08 00:51:38.137489] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF61-0-0] [lt=47] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326298254297, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:38.143257] INFO [STORAGE] normal_running_env_check_ (ob_tenant_snapshot_service.cpp:279) [50474][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=29] normal_running_env_check_ succ(this={is_inited_:true, is_running_:true, running_mode_:3, meta_loaded_:true, tg_id_:391}) [2025-01-08 00:51:38.143287] INFO [STORAGE] try_gc_tenant_snapshot_ (ob_tenant_snapshot_service.cpp:650) [50474][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=29] try_gc_tenant_snapshot finished(ret=0, ret="OB_SUCCESS", this={is_inited_:true, is_running_:true, running_mode_:3, meta_loaded_:true, tg_id_:391}) [2025-01-08 00:51:38.145040] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:830) [50474][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=12] ObTenantSnapshotService thread idle time(tg_id_=391, idle_time=60000, running_mode_=3) [2025-01-08 00:51:38.145576] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:38.147801] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:38.147845] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:38.147928] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:38.147946] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:38.154013] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=26] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:38.154753] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:38.154779] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5764, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:38.155606] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=26] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:38.156659] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC8-0-0] [lt=19] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326298274042, keepalive_ts:-1}) [2025-01-08 00:51:38.156688] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC8-0-0] [lt=27] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326298274042, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:38.167343] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:38.167562] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9861) [2025-01-08 00:51:38.176974] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1736326288171648, end_time=1736326298176971, dag_status={}) [2025-01-08 00:51:38.177024] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=46] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:38.177053] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:38.177062] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:38.177070] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:38.177077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:38.177124] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=46] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:38.177132] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:38.177138] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:38.177144] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:38.177150] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177248] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=96] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177263] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177271] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177278] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177285] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177293] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177368] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177382] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177390] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177398] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177446] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=47] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177456] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177464] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177472] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177480] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177488] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177495] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177502] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177510] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177517] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177524] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177535] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177547] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177555] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177562] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177569] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177576] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177584] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177595] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177603] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177629] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177640] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177648] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177656] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177664] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177672] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177718] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=45] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177729] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177737] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177744] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177771] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177782] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177790] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177798] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177805] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177812] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177819] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177833] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177841] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177866] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=24] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177877] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177885] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177893] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177900] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177908] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177916] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177923] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177931] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177965] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=33] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177973] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177980] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.177987] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178014] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=26] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178025] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178033] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178041] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178070] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=29] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178078] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178085] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178092] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178099] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178106] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:38.178115] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:38.178122] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:38.178128] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:38.178138] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:38.178152] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:38.178158] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:38.178164] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:38.178169] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:38.178174] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:38.178229] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=55] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:38.178241] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:38.185848] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.208324] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=12] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:38.234390] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F8529D-0-0] [lt=23] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:38.237744] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4765-0-0] [lt=95] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:38.240966] INFO [SQL] check_session_leak (ob_sql_session_mgr.cpp:657) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=51] get current session count(used_session_count=12, hold_session_count=12, session_leak_count_threshold=100) [2025-01-08 00:51:38.250641] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3513, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:38.250829] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=185] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:38.250867] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=318, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:38.250961] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=121] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:38.252467] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8830-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:38.252630] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8830-0-0] [lt=161] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5213, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:38.252740] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:38.263455] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1736326298262989000, start_id=1736326298262989000, end_id=1736326298262989001) [2025-01-08 00:51:38.263602] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=62, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:38.263643] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=9272, avg_group_log_size=662) [2025-01-08 00:51:38.263663] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9272) [2025-01-08 00:51:38.265351] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=2068, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:38.265443] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=83] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=484, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:38.265485] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=18, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:38.266077] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E25-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=79, avg_log_submit_to_flush_time=2598, avg_log_submit_to_slide_time=3758) [2025-01-08 00:51:38.266195] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9272, hit_cnt=14, read_cnt=14, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:38.283940] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:38.285906] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.319135] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.320314] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.336308] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:38.336444] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=128] temporary file module use no blocks [2025-01-08 00:51:38.336457] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:38.336471] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:38.336485] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193802249646, flush_io_finished_ret_:0, flush_io_finished_round_:1961, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:38.338032] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4767-0-0] [lt=15][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18309, log_task={header:{begin_lsn:{lsn:2228461}, end_lsn:{lsn:2228583}, log_id:18309, min_scn:{val:1736326297835365003, v:0}, max_scn:{val:1736326297835365003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2228339}, prev_proposal_id:1, committed_end_lsn:{lsn:2228461}, data_checksum:1715074492, accum_checksum:3725463762, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326298237712, freeze_ts:1736326298237712, submit_ts:1736326298237729, flushed_ts:1736326298240136, gen_to_freeze cost time:0, gen_to_submit cost time:17, submit_to_flush cost time:2407}, fs_cb_begin_ts=1736326298338007, log_life_time=100295) [2025-01-08 00:51:38.340276] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=93] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:38.345806] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:38.352075] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58402][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8833-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=20, avg_log_gen_to_submit_time=130, avg_log_submit_to_flush_time=4449, avg_log_submit_to_slide_time=5750) [2025-01-08 00:51:38.352345] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:38.356493] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EF3-0-0] [lt=16] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:38.365368] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C69F-0-0] [lt=31][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193803279201, cache_obj->added_lc()=false, cache_obj->get_object_id()=13659, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:38.366942] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:51:38.367075] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=40] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:51:38.367264] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:38.367853] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=64] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:38.367971] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=4159488) [2025-01-08 00:51:38.368030] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=629149696, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:51:38.368078] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=46] === Run print tenant memory usage task === [2025-01-08 00:51:38.368125] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=14] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,350,656 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 631,246,848 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,298,362,848 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 160,172,020 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,044,885,504 max_mem_memstore_can_get_now= 800,860,136 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,298,362,848 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 70,914,440 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 649,809,920 max_mem_memstore_can_get_now= 354,572,248 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,298,362,848 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 240,672,760 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,114,390,528 max_mem_memstore_can_get_now= 1,203,363,818 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:51:38.368531] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=8] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,657,314,304 total_hold= 3,856,662,528 used= 3,469,381,632 freelists_hold= 187,932,672 total_maps= 1,635 total_unmaps= 243 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,350,656 virtual_memory_used= 4,828,954,624 [CHUNK_MGR] 2 MB_CACHE: hold= 85,983,232 count= 41 pushes= 1,832,622 pops= 1,832,581 maps= 1,504 unmaps= 238 [CHUNK_MGR] 4 MB_CACHE: hold= 7,745,536 count= 3 pushes= 6,154 pops= 6,151 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,128 pops= 3,124 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:51:38.368580] INFO print (ob_malloc_time_monitor.cpp:37) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=32] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 4292, delta_count= 2, avg_cost_time= 2146 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-01-08 00:51:38.368650] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:38.368715] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:38.368894] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:38.371514] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=75][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:38.371917] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=46][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:38.371950] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=18][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:38.378532] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=48] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:38.384273] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1915, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:38.384306] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=368, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:38.384374] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=64] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:38.386362] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.406240] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC8-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:38.406277] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC8-0-0] [lt=34] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:38.418036] INFO [LIB] stat (utility.h:1210) [58538][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A09-0-0] [lt=51] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:38.418116] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58538][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A09-0-0] [lt=68] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:38.418138] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58538][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A09-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:38.419355] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2041, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:38.419386] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:38.419428] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=153, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:38.419547] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=68, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:38.421156] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=102] PNIO [ratelimit] time: 1736326298421153, bytes: 879662272, bw: 0.194152 MB/s, add_ts: 1004990, add_bytes: 204599 [2025-01-08 00:51:38.438392] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4769-0-0] [lt=195] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:38.438463] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4769-0-0] [lt=49] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:38.440165] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2119, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:38.441914] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D978-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:38.441938] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D978-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38089, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:38.441949] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58572][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D978-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=23, avg_log_submit_to_flush_time=2262, avg_log_submit_to_slide_time=38066) [2025-01-08 00:51:38.442023] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=96] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:38.442060] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18219, miss_cnt:0, cache_read_size:2228827, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:38.450947] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=211] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4133, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:38.450990] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:38.451015] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=288, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:38.451132] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=310] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=34, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:38.459207] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.459251] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.459325] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:51:38.484013] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3861) [2025-01-08 00:51:38.484061] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:38.485247] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3861) [2025-01-08 00:51:38.485295] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3861) [2025-01-08 00:51:38.486472] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.486850] INFO [LIB] stat (utility.h:1210) [58565][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CA3-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:38.486872] INFO [LIB] stat (utility.h:1210) [58565][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CA3-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=4456, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:38.486882] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58565][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CA3-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=232, avg_log_gen_to_submit_time=307, avg_log_submit_to_flush_time=2307, avg_log_submit_to_slide_time=4149) [2025-01-08 00:51:38.486969] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3861, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:38.519437] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.541467] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=187] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:38.541526] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=130, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:38.541668] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=69, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:38.542370] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.544063] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.546017] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:38.569385] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=100] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:38.583405] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=67, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:38.586717] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.666648] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46506][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BED1-0-0] [lt=96] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:38.686693] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BB3-0-0] [lt=15] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:38.686830] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.693902] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CA5-0-0] [lt=68] init point thread id with(&point=0x5572d50e5d40, idx_=1232, point=[thread id=58576, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58576) [2025-01-08 00:51:38.741773] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60F-0-0] [lt=29] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:38.741837] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB60F-0-0] [lt=46] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326298324696003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326298324696003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:38.746232] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:38.746739] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58546][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB5A-0-0] [lt=26] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:38.746835] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58546][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB5A-0-0] [lt=88] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:38.749170] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:38.749306] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=99] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, bw:40825, [delay/us]:prepare:7, schedule:64, submit:103, rt:1754, total:1929) [2025-01-08 00:51:38.749324] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:38.749378] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40833, [delay/us]: prepare:7, schedule:64, submit:103, device:1754, total:1929) [2025-01-08 00:51:38.749391] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:38.749429] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4726, iops:13, bw:61251, [delay/us]:prepare:7, schedule:428, submit:398, rt:1454, total:2288) [2025-01-08 00:51:38.749459] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:38.749506] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4726, avg_iops:13, avg_bw:61249, [delay/us]: prepare:7, schedule:428, submit:398, device:1454, total:2288) [2025-01-08 00:51:38.749535] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:38.749560] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:7, bw:28583, [delay/us]:prepare:15, schedule:69, submit:1, rt:1755, total:1841) [2025-01-08 00:51:38.749583] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:171501, [delay/us]:prepare:14, schedule:98, submit:255, rt:1873, total:2240) [2025-01-08 00:51:38.749592] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12250, [delay/us]:prepare:2, schedule:125, submit:110, rt:344, total:581) [2025-01-08 00:51:38.749601] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:38.749635] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:7, avg_bw:28583, [delay/us]: prepare:15, schedule:69, submit:1, device:1755, total:1841) [2025-01-08 00:51:38.749674] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12249, [delay/us]: prepare:2, schedule:125, submit:110, device:344, total:581) [2025-01-08 00:51:38.749684] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:171495, [delay/us]: prepare:14, schedule:98, submit:255, device:1873, total:2240) [2025-01-08 00:51:38.749694] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:38.749715] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4083, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:242, total:248) [2025-01-08 00:51:38.749740] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:38.749770] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4083, [delay/us]: prepare:0, schedule:0, submit:600000000, device:242, total:248) [2025-01-08 00:51:38.749788] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:38.749802] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:38.749811] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:38.749818] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:38.749826] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=228kB/s, net_out=201kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:38.758173] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:62, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2036655524, recv_hp_rpc_cnt:5234, recv_np_rpc_cnt:104080, recv_lp_rpc_cnt:0, recv_mysql_cnt:9377, recv_task_cnt:3750, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10851702, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=116573 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1846, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:94206, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736326298095688}{group_id:10, queue_size:0, recv_req_cnt:11224, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2092, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326289381035}{group_id:2, queue_size:0, recv_req_cnt:30186, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11920, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:4986, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:26842, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=293 pcode=0x1502:cnt=198 pcode=0x717:cnt=195 pcode=0x710:cnt=67 pcode=0x150b:cnt=60, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:51:38.759035] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=859] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:15806811, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8220, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3590716, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=17, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:51:38.759660] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=622] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:15386370, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3577290, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:51:38.760246] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=584] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:24, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:884914703, recv_hp_rpc_cnt:3756, recv_np_rpc_cnt:38190, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:783683, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=65879 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:68271, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:7, nesting_worker_cnt:0, token_change:1736326297966735}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2202, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326279406832}{group_id:2, queue_size:0, recv_req_cnt:15581, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18681, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=306 pcode=0x51c:cnt=246 pcode=0x717:cnt=185 pcode=0x710:cnt=99 pcode=0x150b:cnt=60, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:51:38.760991] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=741] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:59, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9692565987, recv_hp_rpc_cnt:77, recv_np_rpc_cnt:35174, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2963924, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=25176 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:0, recv_req_cnt:7064, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326269316514}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:151283, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:7, nesting_worker_cnt:0, token_change:1736326297943423}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326242277705}{group_id:19, queue_size:0, recv_req_cnt:410, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326264650105}{group_id:2, queue_size:0, recv_req_cnt:37376, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:11175, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=396 pcode=0x1501:cnt=200 pcode=0x1516:cnt=191 pcode=0x717:cnt=190 pcode=0x150b:cnt=120, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:51:38.764748] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1736326298324696003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3, epoch_tstamp:1736326298764644}, version_delta=440051) [2025-01-08 00:51:38.770105] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=57] clear dtl interm result cost(us)(clear_cost=5111, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:38.770438] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=22] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43436746}, readable_log_tail:{lsn:43436746}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23668022, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:38.770529] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=89] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43436746}, readable_log_tail:{lsn:43436746}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23668022, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:38.770601] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=70] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:38.770637] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=14] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:38.770931] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:38.771166] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] clear dtl interm result cost(us)(clear_cost=13, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:38.780061] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=46] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:38.792735] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=98] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:38.792804] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=11] ====== tenant freeze timer task ====== [2025-01-08 00:51:38.793170] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.796223] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=147] BlockGCTimerTask success(ret=0, cost_time_us=53, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:38.800229] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:38.802249] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=119] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:38.802339] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326294359805000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:38.803793] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84392065}], scn_array=[cnt:1, {val:1736326298782784000, v:0}], curr_size=586, accum_size=1293, time_cost=20426) [2025-01-08 00:51:38.806259] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=50] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8751, offset=2485133}, end_cursor:ObLogCursor{file_id=1, log_id=8752, offset=2485438}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bac050, len:1139, log_data_len:305, seq:8751, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:38.809688] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3F5-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:38.809874] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=31] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:38.809929] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E66-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:38.809963] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E66-0-0] [lt=15] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326298339955004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326298339955004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:38.812266] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:38.812348] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:38.814230] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8711B-0-0] [lt=9][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:38.814270] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:38.814342] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:38.814549] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=98] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:38.814840] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=18] gc stale ls task succ [2025-01-08 00:51:38.814874] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:38.814907] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=9] gc stale ls task succ [2025-01-08 00:51:38.816135] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=40] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1}, {id:1003}]) [2025-01-08 00:51:38.819296] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D08F-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:38.825907] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=19] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:38.827348] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8E6-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1736326298819805000, start_id=1736326298819805000, end_id=1736326298819805001) [2025-01-08 00:51:38.827360] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=47] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:38.827505] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=72] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326295661593000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:38.828977] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=45] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8752, offset=2485438}, end_cursor:ObLogCursor{file_id=1, log_id=8753, offset=2485735}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bb4050, len:834, log_data_len:297, seq:8752, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:38.829786] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E66-0-0] [lt=21] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:38.829854] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E66-0-0] [lt=46] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326298339955004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326298339955004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:38.838829] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F57C-0-0] [lt=116] get number(ret=0, service_type_=0, range=1, base_id=1736326298838711000, start_id=1736326298838711000, end_id=1736326298838711001) [2025-01-08 00:51:38.840374] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=50] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8753, offset=2485735}, end_cursor:ObLogCursor{file_id=1, log_id=8754, offset=2486032}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bb8050, len:537, log_data_len:297, seq:8753, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:38.843389] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8754, offset=2486032}, end_cursor:ObLogCursor{file_id=1, log_id=8755, offset=2486329}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bbc050, len:4336, log_data_len:297, seq:8754, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:38.843840] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=115] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18121, miss_cnt:0, cache_read_size:2216672, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:38.851857] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:38.857188] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918543D-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:38.857517] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:38.857540] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-01-08 00:51:38.857549] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:38.857562] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:38.857627] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=63] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193802769234, flush_io_finished_ret_:0, flush_io_finished_round_:1957, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:38.857670] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:38.857687] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-01-08 00:51:38.857694] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:38.857702] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:38.857712] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193802769314, flush_io_finished_ret_:0, flush_io_finished_round_:3785, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:38.866288] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AE4F-0-0] [lt=51] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.840757417678833008e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=392, Throttle Trigger(MB)=235, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.429380301347038085e+02) [2025-01-08 00:51:38.872976] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=18] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:38.874263] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=44] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:38.874372] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=38] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326295766421000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:38.874490] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=74] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:38.874580] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:38.874688] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=17] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:38.874862] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=93] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:38.875124] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:38.878831] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:38.893348] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=94][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:38.896538] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.896834] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.897366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.897485] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:38.915090] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:38.938601] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=89] PNIO [ratelimit] time: 1736326298938597, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007587, add_bytes: 0 [2025-01-08 00:51:38.949459] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=699] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:38.950906] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=42] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1399792, replace_num=15728) [2025-01-08 00:51:38.951001] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=93] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:38.951331] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58518][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=14] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:38.954588] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE4-0-0] [lt=26] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:38.954640] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE4-0-0] [lt=44] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=55) [2025-01-08 00:51:38.963268] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=95] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:38.970002] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:38.971514] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:38.971563] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58542][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=64] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:51:38.972951] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=34] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1100995, clean_num=31457) [2025-01-08 00:51:38.984693] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=4323, avg_group_log_size=480) [2025-01-08 00:51:38.993505] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=98][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.022043] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A947B-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:39.022074] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A947B-0-0] [lt=30] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20286, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:39.022085] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A947B-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=22, avg_log_submit_to_flush_time=2217, avg_log_submit_to_slide_time=20264) [2025-01-08 00:51:39.022110] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:39.041294] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4775-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18316, log_task={header:{begin_lsn:{lsn:2229315}, end_lsn:{lsn:2229437}, log_id:18316, min_scn:{val:1736326298838711001, v:0}, max_scn:{val:1736326298838711001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2229193}, prev_proposal_id:1, committed_end_lsn:{lsn:2229315}, data_checksum:1715074492, accum_checksum:923907384, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326298941084, freeze_ts:1736326298941084, submit_ts:1736326298941092, flushed_ts:1736326298942639, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:1547}, fs_cb_begin_ts=1736326299041263, log_life_time=100179) [2025-01-08 00:51:39.041476] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=93] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=50, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:39.041502] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:39.050433] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=79, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:39.050481] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:39.050593] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=51] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=46, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:39.050627] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:39.050967] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=35] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:288K/66:436634 disk:288K/66:436634 t1m2:36K/8:54909 t1m2g0:36K/8:54909 t1002m2:188K/44:562407 t1002m2g0:188K/44:562407 t1001m2:64K/14:259476 t1001m2g0:64K/14:259476 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:39.051902] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:39.067850] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=103] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:39.075930] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB1-0-0] [lt=56][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:39.083566] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E30-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9864) [2025-01-08 00:51:39.083647] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E30-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=9, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:39.083662] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E30-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4948, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:39.086027] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326298324696004, v:0}, new_max_version={val:1736326298324696004, v:0}, last_min_version={val:1736326298324696002, v:0}, last_max_version={val:1736326298324696002, v:0}, total_time_us=20541, delta_us=761328) [2025-01-08 00:51:39.086055] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6A1-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193803999889, cache_obj->added_lc()=false, cache_obj->get_object_id()=13660, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:39.086063] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326298324696004, v:0}, min_version={val:1736326298324696004, v:0}, max_version={val:1736326298324696004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:39.088006] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB1-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:39.091517] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5282807193, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:39.091572] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:39.091587] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8df7f30f0) [2025-01-08 00:51:39.091596] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5282807193, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:39.093799] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.097491] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:39.097950] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58565][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:51:39.150588] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:39.150698] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=105] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:39.150784] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:39.150800] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:39.151283] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=60] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:39.152219] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:39.168381] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B9-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:39.168418] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2B9-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326298838711003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326298838711003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:39.171356] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1736326299168662000, start_id=1736326299168662000, end_id=1736326299168662001) [2025-01-08 00:51:39.174092] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:39.177046] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326289171760, end_time=1736326299177044, dag_status={}) [2025-01-08 00:51:39.177077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:39.177090] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:39.177098] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:39.177106] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:39.177114] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:39.177149] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=34] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:39.177157] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:39.177164] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:39.177223] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:39.177232] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177243] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177252] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177260] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177268] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177276] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177284] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177292] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177300] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177309] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177316] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177332] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177340] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177349] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177357] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177365] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177373] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177381] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177389] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177397] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177412] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177420] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177428] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177436] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177444] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177504] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177514] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177522] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177530] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177538] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177593] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=53] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177602] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177610] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177618] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177626] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177634] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177642] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177650] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177658] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177666] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177674] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177682] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177690] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177698] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177706] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177713] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177721] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177729] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177736] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177744] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177753] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177760] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177768] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177776] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177784] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177792] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177799] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177808] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177815] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177823] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177831] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177839] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177847] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177877] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=30] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177889] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177897] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177905] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177958] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=52] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177971] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177979] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177987] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:39.177997] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:39.178004] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:39.178011] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:39.178016] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:39.178022] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:39.178027] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:39.178032] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:39.178037] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:39.178042] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:39.178047] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:39.178053] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:39.178136] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=8525) [2025-01-08 00:51:39.193159] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-01-08 00:51:39.193254] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=90] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:39.193612] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:39.194021] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.237217] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=44] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:39.241373] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4779-0-0] [lt=61] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:39.253890] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2560, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:39.253966] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:39.253990] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=270, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:39.254050] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:39.254231] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9483-0-0] [lt=50] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:39.254255] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9483-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4196, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:39.254716] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:39.256713] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:39.256740] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4551, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:39.267629] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=63, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:39.267668] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=9275, avg_group_log_size=662) [2025-01-08 00:51:39.267687] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9275) [2025-01-08 00:51:39.269290] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=2562, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:39.269334] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=522, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:39.270707] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=16, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:39.275360] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E33-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=89, avg_log_submit_to_flush_time=3922, avg_log_submit_to_slide_time=5624) [2025-01-08 00:51:39.275815] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=106] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9275, hit_cnt=14, read_cnt=14, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:39.283782] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=33] start to print blacklist info [2025-01-08 00:51:39.284296] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=29] ls blacklist refresh finish(cost_time=26387) [2025-01-08 00:51:39.284318] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:39.284733] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=52] ====== check clog disk timer task ====== [2025-01-08 00:51:39.284759] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=21] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:39.294422] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.319272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.320372] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=175] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.325931] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:39.326272] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%", tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-01-08 00:51:39.327960] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8E9-0-0] [lt=88] get number(ret=0, service_type_=0, range=1, base_id=1736326299327803000, start_id=1736326299327803000, end_id=1736326299327803001) [2025-01-08 00:51:39.336645] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:39.336679] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-01-08 00:51:39.336690] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:39.336704] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:39.336717] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193803250359, flush_io_finished_ret_:0, flush_io_finished_round_:1962, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:39.338346] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1829F57E-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326299338309000, start_id=1736326299338309000, end_id=1736326299338309001) [2025-01-08 00:51:39.342108] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=47] ts response task statistics(alloc_count=60554, free_count=30276) [2025-01-08 00:51:39.342166] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=47] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:39.342665] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=497, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:39.343477] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=38] blacklist_loop exec finished(cost_time=30, is_enabled=true, send_cnt=0) [2025-01-08 00:51:39.351658] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:39.353980] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58560][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9485-0-0] [lt=22] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=91, avg_log_submit_to_flush_time=3149, avg_log_submit_to_slide_time=4271) [2025-01-08 00:51:39.354144] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:39.370794] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D49-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:39.374646] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=38] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:39.375109] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=71] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:39.375600] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:39.375739] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:39.375916] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:39.376337] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:39.378918] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:39.380510] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC2-0-0] [lt=31] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:39.380619] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC2-0-0] [lt=96] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:39.380800] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-01-08 00:51:39.390155] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6A3-0-0] [lt=98][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193804303985, cache_obj->added_lc()=false, cache_obj->get_object_id()=13661, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:39.393803] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:39.393835] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:39.393848] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:39.393859] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:39.396356] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.396630] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:39.397426] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=790] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:39.399314] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=3887, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:39.399411] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=71] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=165, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:39.399533] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=46, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:39.408070] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC9-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:39.408234] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CC9-0-0] [lt=102] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:39.414290] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB65-0-0] [lt=29] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:39.414517] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB65-0-0] [lt=215] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:39.420535] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A1D-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:39.420617] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A1D-0-0] [lt=68] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:39.420638] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A1D-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:39.422020] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2050, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:39.422162] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=139] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:39.422308] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=136] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=106, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:39.422411] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=75, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:39.428737] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=81] PNIO [ratelimit] time: 1736326299428734, bytes: 879923328, bw: 0.247089 MB/s, add_ts: 1007581, add_bytes: 261056 [2025-01-08 00:51:39.441627] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A477D-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:39.441745] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A477D-0-0] [lt=102] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:39.442885] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1599, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:39.445650] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8853-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:39.445696] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8853-0-0] [lt=45] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12885, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:39.445710] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8853-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=1809, avg_log_submit_to_slide_time=12864) [2025-01-08 00:51:39.445892] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:39.445927] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18229, miss_cnt:0, cache_read_size:2230047, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:39.452732] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2634, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:39.452760] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:39.452793] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=284, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:39.452847] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:39.453975] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2200141, this={dio_aligned_buf:{buf_write_offset:711, buf_padding_size:3385, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:22906, truncate_used_ts:5187}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2199552, buf_len=122, total_write_size=11928, total_write_size_after_dio=413696, ob_pwrite_used_ts=277121, count=98, each_pwrite_cost=2827) [2025-01-08 00:51:39.459065] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.459104] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=57] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.459508] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=27] gc stale ls task succ [2025-01-08 00:51:39.463684] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=38] LogLoopThread round_cost_time(us)(round_cost_time=14) [2025-01-08 00:51:39.463786] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=90] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:44897, max_lsn:84394652, max_scn:{val:1736326299381595000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84394652}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42451612}}, last_submit_lsn:{lsn:84394066}, last_submit_end_lsn:{lsn:84394652}, last_submit_log_id:44897, last_submit_log_pid:1, max_flushed_lsn:{lsn:84394066}, max_flushed_end_lsn:{lsn:84394652}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84394652}, last_slide_log_id:44897, last_slide_scn:{val:1736326299381595000, v:0}, last_slide_lsn:{lsn:84394066}, last_slide_end_lsn:{lsn:84394652}, last_slide_log_pid:1, last_slide_log_accum_checksum:2562442373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:44898, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84394652}, readable_log_tail:{lsn:84394652}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49814884, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:51:39.464246] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=396] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326299399501, last_flushed_end_lsn:{lsn:84394652}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326299395847, last_flushed_end_lsn:{lsn:84394652}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326299405390, last_flushed_end_lsn:{lsn:84394652}}]) [2025-01-08 00:51:39.464274] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=44898, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326299399501, last_flushed_end_lsn:{lsn:84394652}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326299395847, last_flushed_end_lsn:{lsn:84394652}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326299405390, last_flushed_end_lsn:{lsn:84394652}}]) [2025-01-08 00:51:39.466388] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326298819805004, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=646574, in_cluster_service=true, cluster_version={val:1736326298819805001, v:0}, min_cluster_version={val:1736326298819805001, v:0}, max_cluster_version={val:1736326298819805001, v:0}, get_cluster_version_err=0, cluster_version_delta=646574, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18528, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=527383906, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:39.473561] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7] LogLoopThread round_cost_time(us)(round_cost_time=68) [2025-01-08 00:51:39.474301] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=30] ====== tenant freeze timer task ====== [2025-01-08 00:51:39.489128] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] ====== tenant freeze timer task ====== [2025-01-08 00:51:39.489257] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=116] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71124160, tenant_id=1001) [2025-01-08 00:51:39.489286] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:39.493953] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3416) [2025-01-08 00:51:39.494037] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=78] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:39.495980] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3416) [2025-01-08 00:51:39.496167] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3416) [2025-01-08 00:51:39.496510] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.498106] INFO [LIB] stat (utility.h:1210) [58563][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6944-0-0] [lt=35] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=8, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:39.498128] INFO [LIB] stat (utility.h:1210) [58563][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6944-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=5711, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:39.498139] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58563][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6944-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=125, avg_log_submit_to_flush_time=4214, avg_log_submit_to_slide_time=5586) [2025-01-08 00:51:39.498551] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3416, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:39.519623] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.520675] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=44] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:39.542279] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.543294] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:39.543331] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=131, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:39.543414] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=63] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=83, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:39.543801] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.552072] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:39.558442] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAECD-0-0] [lt=36] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:39.558484] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAECD-0-0] [lt=33] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:39.575628] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:39.577261] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=84] get_max_applied_scn(scn={val:1736326299468425000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43445431}, palf_committed_end_scn_:{val:1736326299572894000, v:0}, last_check_scn_:{val:1736326299572894000, v:0}, max_applied_cb_scn_:{val:1736326299468425000, v:0}}) [2025-01-08 00:51:39.583772] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:51:39.594087] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=99] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=92, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:39.594231] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:39.594249] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] try gc retain ctx [2025-01-08 00:51:39.594257] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-01-08 00:51:39.594285] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1736326299381595000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84394774}, palf_committed_end_scn_:{val:1736326299381595001, v:0}, last_check_scn_:{val:1736326299590771000, v:0}, max_applied_cb_scn_:{val:1736326299381595000, v:0}}) [2025-01-08 00:51:39.594315] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84394774}, readable_log_tail:{lsn:84394774}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49814762, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:39.594398] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=79] finish update min start scn(max_decided_scn={val:1736326299381595000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326294358680000, v:0}, keep_alive_scn_:{val:1736326294559167001, v:0}, update_ts_:1736326299590771}) [2025-01-08 00:51:39.595250] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309254996000, latest_log_ts:{val:1736326292647912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:39.595585] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=89] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:39.595619] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=32] try gc retain ctx [2025-01-08 00:51:39.595631] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-01-08 00:51:39.595668] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] finish update min start scn(max_decided_scn={val:1736326299572894000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326289514488001, v:0}, keep_alive_scn_:{val:1736326294544299001, v:0}, update_ts_:1736326299590771}) [2025-01-08 00:51:39.597271] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309284252000, latest_log_ts:{val:1736326292617740001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:39.597845] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.601019] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:39.647008] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58579][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:39.647300] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:39.697999] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:39.698035] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.710858] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=27] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:39.721846] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=43] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:39.721887] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=41] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:39.721943] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498D-0-0] [lt=28] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:39.721962] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498D-0-0] [lt=15] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:39.751368] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:39.751453] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=49] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44959, [delay/us]:prepare:7, schedule:50, submit:716, rt:3077, total:3853) [2025-01-08 00:51:39.751469] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:39.751557] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44958, [delay/us]: prepare:7, schedule:50, submit:716, device:3077, total:3853) [2025-01-08 00:51:39.751574] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:39.751664] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:15, bw:69480, [delay/us]:prepare:14, schedule:118, submit:695, rt:1153, total:1982) [2025-01-08 00:51:39.751677] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:39.751716] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:15, avg_bw:69478, [delay/us]: prepare:14, schedule:118, submit:695, device:1153, total:1982) [2025-01-08 00:51:39.751750] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:39.751776] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20434, [delay/us]:prepare:5, schedule:2085, submit:1, rt:3008, total:5100) [2025-01-08 00:51:39.751799] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:171651, [delay/us]:prepare:13, schedule:412, submit:375, rt:1105, total:1906) [2025-01-08 00:51:39.751809] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:39.751838] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20434, [delay/us]: prepare:5, schedule:2085, submit:1, device:3008, total:5100) [2025-01-08 00:51:39.751853] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:171659, [delay/us]: prepare:13, schedule:412, submit:375, device:1105, total:1906) [2025-01-08 00:51:39.751884] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:39.751907] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4087, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:341, total:350) [2025-01-08 00:51:39.751992] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=66] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:39.752027] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4086, [delay/us]: prepare:0, schedule:0, submit:600000000, device:341, total:350) [2025-01-08 00:51:39.752051] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:39.752085] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:39.752095] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:39.752103] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:39.752110] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=224kB/s, net_out=229kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:39.752506] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=216] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:39.765364] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085167-0-0] [lt=25] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:39.765530] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085169-0-0] [lt=61] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:39.766312] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:39.768163] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326299767653000, start_id=1736326299767653000, end_id=1736326299767653001) [2025-01-08 00:51:39.772097] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18715277, this={dio_aligned_buf:{buf_write_offset:1198, buf_padding_size:2898, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:32739, truncate_used_ts:6749}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=18714624, buf_len=545, total_write_size=15408, total_write_size_after_dio=425984, ob_pwrite_used_ts=302051, count=100, each_pwrite_cost=3020) [2025-01-08 00:51:39.776119] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:39.778288] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=75] success to update min reserved snapshot(reserved_snapshot=1736324495661593000, duration=1800, snapshot_gc_ts=1736326295661593000) [2025-01-08 00:51:39.784277] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=36] update reserved snapshot finished(cost_ts=5991, reserved_snapshot=1736324495661593000) [2025-01-08 00:51:39.786746] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB610-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:39.786792] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB610-0-0] [lt=31] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326299327803003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326299327803003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:39.799635] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.809379] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50623][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB610-0-0] [lt=5] get number(ret=0, service_type_=0, range=1, base_id=1736326299809340000, start_id=1736326299809340000, end_id=1736326299809340001) [2025-01-08 00:51:39.812654] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:39.812706] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:39.813050] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=34] ====== check clog disk timer task ====== [2025-01-08 00:51:39.813276] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=219] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:39.815089] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:39.815747] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:39.815817] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:39.815862] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=8] gc stale ls task succ [2025-01-08 00:51:39.816319] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1}, {id:1003}]) [2025-01-08 00:51:39.816482] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=12] finish to resize log external storage handler(current_ts=1736326299816478, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:39.816502] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=19] snapshot_traffic_info once [2025-01-08 00:51:39.818380] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:39.818521] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%", tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:39.821104] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3F8-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:39.823704] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D092-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:39.824115] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87121-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:39.824178] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:39.824192] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:39.824847] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=28] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:39.826412] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=39] ====== checkpoint timer task ====== [2025-01-08 00:51:39.827413] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:39.827455] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=41] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:39.827559] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=24] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309284252000, latest_log_ts:{val:1736326292617740001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:39.827638] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=75] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:39.827667] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:39.827681] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326292617740001, v:0}) [2025-01-08 00:51:39.827689] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:39.827696] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:39.827708] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:39.827725] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:39.827737] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:39.827756] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:39.827777] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:39.827786] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:39.843225] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1736326299843057000, start_id=1736326299843057000, end_id=1736326299843057001) [2025-01-08 00:51:39.843290] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=50] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326299843156], gts:1736326299843057000, latest_srr:[mts=1736326299843156]}) [2025-01-08 00:51:39.843324] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326299843313], gts:1736326299843057000, latest_srr:[mts=1736326299843313]}) [2025-01-08 00:51:39.843599] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326299843585], gts:1736326299843057000, latest_srr:[mts=1736326299843585]}) [2025-01-08 00:51:39.852089] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=87] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:39.858429] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:39.858484] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=46] temporary file module use no blocks [2025-01-08 00:51:39.858496] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:39.858510] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:39.858524] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193803771575, flush_io_finished_ret_:0, flush_io_finished_round_:3786, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:39.860404] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918543E-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:39.860812] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:39.860837] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-01-08 00:51:39.860847] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:39.860859] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:39.860872] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193803771501, flush_io_finished_ret_:0, flush_io_finished_round_:1958, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:39.862153] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=60] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8755, offset=2486329}, end_cursor:ObLogCursor{file_id=1, log_id=8756, offset=2486634}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bc0050, len:4039, log_data_len:305, seq:8755, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:39.865003] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E67-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:39.865064] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E67-0-0] [lt=34] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326299341876004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326299341876004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:39.869377] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AE5E-0-0] [lt=103] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=107) [2025-01-08 00:51:39.874461] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326299668627000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=205825, in_cluster_service=true, cluster_version={val:1736326299572894000, v:0}, min_cluster_version={val:1736326299572894000, v:0}, max_cluster_version={val:1736326299572894000, v:0}, get_cluster_version_err=0, cluster_version_delta=301558, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18965, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=645511235, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:39.874949] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=82] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:39.876017] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=103] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:39.876065] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=43] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:39.876250] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=79] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:39.876342] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=12] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:39.877801] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326299875408000, start_id=1736326299875408000, end_id=1736326299875408001) [2025-01-08 00:51:39.879058] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=113] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:39.879399] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E67-0-0] [lt=22] get number(ret=0, service_type_=0, range=1, base_id=1736326299878310000, start_id=1736326299878310000, end_id=1736326299878310001) [2025-01-08 00:51:39.882727] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:39.889453] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=86] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8756, offset=2486634}, end_cursor:ObLogCursor{file_id=1, log_id=8757, offset=2486931}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bc4050, len:3734, log_data_len:297, seq:8756, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:39.891694] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E67-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:39.891795] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E67-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326299843057000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326299843057000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:39.894446] INFO [SQL] alloc_value (ob_sql_session_mgr.cpp:272) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E67-0-0] [lt=23] alloc_session_count(alloc_total_count=360000) [2025-01-08 00:51:39.896513] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E67-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.816756868362426758e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.520464407226368166e+02) [2025-01-08 00:51:39.897475] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.898000] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.898033] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.898826] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:39.900679] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=114][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:39.905914] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=48] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8757, offset=2486931}, end_cursor:ObLogCursor{file_id=1, log_id=8758, offset=2487228}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bc8050, len:3437, log_data_len:297, seq:8757, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:39.906954] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8758, offset=2487228}, end_cursor:ObLogCursor{file_id=1, log_id=8759, offset=2487525}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bcc050, len:3140, log_data_len:297, seq:8758, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:39.907149] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18132, miss_cnt:0, cache_read_size:2218005, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:39.909230] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=16] ====== checkpoint timer task ====== [2025-01-08 00:51:39.910071] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=79] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:39.910139] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=66] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:39.910298] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=27] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309254996000, latest_log_ts:{val:1736326292647912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:39.910404] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=102] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309254996000, latest_log_ts:{val:1736326292647912001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:39.910481] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=70] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:39.910494] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326292647912001, v:0}) [2025-01-08 00:51:39.910505] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:39.910516] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:39.910529] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:39.910566] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=30] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:39.910582] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:39.910599] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:39.910695] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=95] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:39.910709] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:39.915066] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=15] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:39.917039] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1736326299914271000, start_id=1736326299914271000, end_id=1736326299914271001) [2025-01-08 00:51:39.920342] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=18] [APPLY STAT CB APPEND COST TIME](cur_stat_count=55, stat_interval=5000000, avg (count/cost)=122, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:39.920409] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=66] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=55, stat_interval=5000000, avg (count/cost)=5256, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:39.920418] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=55, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:39.920426] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=55, stat_interval=5000000, avg (count/cost)=-901, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:39.920492] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=65] [APPLY STAT CB TOTAL TIME](cur_stat_count=55, stat_interval=5000000, avg (count/cost)=4478, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:39.946254] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1736326299946250, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007653, add_bytes: 0 [2025-01-08 00:51:39.958513] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:39.959329] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=33] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12802318336, fast_delete_log_mode=false) [2025-01-08 00:51:39.973829] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=18] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:39.974270] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:39.976086] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58526][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=19] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:39.976638] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:39.982359] INFO [SQL] free_value (ob_sql_session_mgr.cpp:325) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=109] free_session_count(free_total_count=360000) [2025-01-08 00:51:39.988956] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=98] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:51:39.993830] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4084, avg_group_log_size=371) [2025-01-08 00:51:40.000830] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.023218] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=48] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:40.034033] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D997-0-0] [lt=120] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:40.034067] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D997-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4028, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:40.034079] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D997-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=26, avg_log_submit_to_flush_time=1439, avg_log_submit_to_slide_time=4002) [2025-01-08 00:51:40.034245] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:40.034301] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] succ to submit log task to replay service(to_submit_lsn={lsn:2218183}, to_submit_scn={val:1736326299827945001, v:0}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326300034022, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2218249}, next_to_submit_scn_:{val:1736326299827945002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2218127}, end_lsn:{lsn:2218249}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2218249}, readable_log_tail:{lsn:2218249}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64886519, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326299827945001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326299827945001, v:0}, data_checksum:2124447694, flag:27498}}, init_mode_version:0, accumulate_checksum:586422800, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326299827945001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18134, miss_cnt:0, cache_read_size:2218249, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:40.045718] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=61] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=74, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:40.045769] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=50] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:40.054292] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=69, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:40.054363] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:40.054438] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:40.054461] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:40.054726] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/72:128634 disk:308K/72:128634 t1m2:48K/11:40970 t1m2g0:48K/11:40970 t1002m2:184K/45:178702 t1002m2g0:184K/45:178702 t1001m2:76K/16:48088 t1001m2g0:76K/16:48088 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:40.056556] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:51:40.069973] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:40.098030] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB2-0-0] [lt=60][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:40.098685] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:40.099738] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58563][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:51:40.100615] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB2-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:40.100967] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=94][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.111678] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58581][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:40.111775] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:51:40.146653] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58581][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD90-0-0] [lt=31] init point thread id with(&point=0x5572d50e5e80, idx_=1237, point=[thread id=58581, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58581) [2025-01-08 00:51:40.146748] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58581][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD90-0-0] [lt=88] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326300261897, keepalive_ts:-1}) [2025-01-08 00:51:40.146769] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58581][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD90-0-0] [lt=19] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326300261897, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:40.154401] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1755, avg_group_log_size=159) [2025-01-08 00:51:40.154519] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=112] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1755) [2025-01-08 00:51:40.154660] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:51:40.154678] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1332) [2025-01-08 00:51:40.156531] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=303] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1755) [2025-01-08 00:51:40.158770] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=60] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:40.177746] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:40.179984] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9932) [2025-01-08 00:51:40.180040] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9932) [2025-01-08 00:51:40.182529] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3AD9-0-0] [lt=194] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:40.182613] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3AD9-0-0] [lt=83] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=5174, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:40.183104] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=95] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326299972668000, v:0}, new_max_version={val:1736326299972668000, v:0}, last_min_version={val:1736326299809340000, v:0}, last_max_version={val:1736326299809340000, v:0}, total_time_us=6174, delta_us=210433) [2025-01-08 00:51:40.183152] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326299972668000, v:0}, min_version={val:1736326299972668000, v:0}, max_version={val:1736326299972668000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:40.198389] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BA-0-0] [lt=29] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:40.198502] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BA-0-0] [lt=95] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326299840677002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326299840677002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:40.201166] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.243849] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A478D-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:40.257294] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1989, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:40.257386] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=89] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:51:40.257466] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=462, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:40.257558] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=111] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=83, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:40.259261] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A94A2-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:40.259287] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A94A2-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4017, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:40.259985] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A94A3-0-0] [lt=57] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:40.260010] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A94A3-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4318, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:40.260046] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:40.271236] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=131] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=74, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:40.271296] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9932, avg_group_log_size=620) [2025-01-08 00:51:40.271318] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9932) [2025-01-08 00:51:40.272431] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=1446, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:40.272466] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=402, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:40.272568] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=12, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:40.272816] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58582][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:40.273379] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:40.283105] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58582][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC886F-0-0] [lt=62] init point thread id with(&point=0x5572d50e5ec0, idx_=1238, point=[thread id=58582, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58582) [2025-01-08 00:51:40.284557] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:40.291953] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=26] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:40.301295] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:40.301366] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=69][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.319423] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.319523] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.330301] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE8ED-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1736326300329087000, start_id=1736326300329087000, end_id=1736326300329087001) [2025-01-08 00:51:40.336913] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:40.336967] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=46] temporary file module use no blocks [2025-01-08 00:51:40.336981] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:40.336996] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:40.337010] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193804250589, flush_io_finished_ret_:0, flush_io_finished_round_:1963, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:40.343989] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=42] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:40.357278] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8871-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=106, avg_log_submit_to_flush_time=2703, avg_log_submit_to_slide_time=3875) [2025-01-08 00:51:40.357573] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1755, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:40.359047] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:40.375327] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:40.376238] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=72] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:40.376428] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=39] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:40.376497] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:40.378323] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=91] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:40.379170] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:40.382103] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E44-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=104, avg_log_submit_to_flush_time=1845, avg_log_submit_to_slide_time=4520) [2025-01-08 00:51:40.382291] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10521, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:40.383324] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:40.401598] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.402874] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=89] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2308, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:40.402921] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=322, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:40.403006] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=140] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=37, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:40.404735] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:40.410040] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCA-0-0] [lt=17] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:40.410074] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCA-0-0] [lt=31] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:40.418412] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46499][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6A5-0-0] [lt=29][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193805332244, cache_obj->added_lc()=false, cache_obj->get_object_id()=13662, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:40.432034] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A31-0-0] [lt=214] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:40.432091] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A31-0-0] [lt=30] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:40.432111] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A31-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:40.433333] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=992, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:40.433384] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:40.433408] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=218, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:40.433474] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=51] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=90, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:40.434780] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1736326300434777, bytes: 880176428, bw: 0.239925 MB/s, add_ts: 1006043, add_bytes: 253100 [2025-01-08 00:51:40.443979] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4791-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:40.444010] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4791-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:40.445061] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1456, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:40.446493] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8873-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:40.446521] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8873-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12601, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:40.446531] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8873-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=12, avg_log_submit_to_flush_time=1790, avg_log_submit_to_slide_time=12589) [2025-01-08 00:51:40.446525] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:40.446558] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18239, miss_cnt:0, cache_read_size:2231267, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:40.456746] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2455, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:40.456775] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1755) [2025-01-08 00:51:40.456792] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=171, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:40.457027] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=46, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:40.459373] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.459427] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.459914] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=73] gc stale ls task succ [2025-01-08 00:51:40.485483] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=44] new mysql sessid created(conn.sessid_=3221585668, support_ssl=false) [2025-01-08 00:51:40.485659] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=102] sm conn init succ(conn.sessid_=3221585668, sess.client_addr_="192.168.159.136:39662") [2025-01-08 00:51:40.485681] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x2ad8a0dfc030, session_id:3221585668, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:156}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:40.486499] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:40.486520] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:292) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=21] mysql login queue(mysql_queue->queue_.size()=1) [2025-01-08 00:51:40.486585] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A87-0-0] [lt=24] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221585668, conn.client_addr_port_=39662, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221585668, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221585668, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D17784A87-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46334, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523331686572101, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:80, fixup_index:-1, fixup_ash_buffer:{this:0x2ad8af7caaf8, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:51:40.487924] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A87-0-0] [lt=65][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:51:40.488015] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A87-0-0] [lt=37] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221585668, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:40.490269] INFO [SERVER] process (obmp_init_db.cpp:53) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B8C-0-0] [lt=129] init db(db_name=oceanbase) [2025-01-08 00:51:40.497671] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B90-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12358, add_timestamp:1736326300497603}) [2025-01-08 00:51:40.497721] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B90-0-0] [lt=29] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12396, add_timestamp:1736326300497719}) [2025-01-08 00:51:40.501413] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B90-0-0] [lt=11][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:40.502152] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.502706] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B91-0-0] [lt=18] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11001, add_timestamp:1736326300502702}) [2025-01-08 00:51:40.502746] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B91-0-0] [lt=19][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12396, add_timestamp:1736326300502743}) [2025-01-08 00:51:40.502759] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B91-0-0] [lt=12][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12396, add_timestamp:1736326300502743}) [2025-01-08 00:51:40.502742] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285990-0-0] [lt=17] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12358, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326300502719, sql_renew_time:1736326300502719, is_mark_fail:false}]) [2025-01-08 00:51:40.502777] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285990-0-0] [lt=34] success to process renew task(task={tenant_id:1, table_id:12358, add_timestamp:1736326300497603}, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326300502719, sql_renew_time:1736326300502719, is_mark_fail:false}]) [2025-01-08 00:51:40.502804] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285991-0-0] [lt=16] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12396, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326300502799, sql_renew_time:1736326300502799, is_mark_fail:false}]) [2025-01-08 00:51:40.502819] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285991-0-0] [lt=15] success to process renew task(task={tenant_id:1, table_id:12396, add_timestamp:1736326300497719}, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326300502799, sql_renew_time:1736326300502799, is_mark_fail:false}]) [2025-01-08 00:51:40.502767] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B91-0-0] [lt=8][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=12396, vtable_locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326285363385, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:40.502837] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285992-0-0] [lt=13] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11001, vtable_type={tenant_id:1, type:2}) [2025-01-08 00:51:40.505576] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285992-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11001, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326300505563, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:40.505607] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285992-0-0] [lt=30] success to process renew task(task={tenant_id:1, table_id:11001, add_timestamp:1736326300502702}, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326300505563, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:40.508422] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=118] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4083) [2025-01-08 00:51:40.508450] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:40.509790] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B91-0-0] [lt=73][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:40.511677] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58581][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA694F-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4083) [2025-01-08 00:51:40.512241] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4083) [2025-01-08 00:51:40.512871] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:40.512910] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4861, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:40.512923] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=216, avg_log_gen_to_submit_time=322, avg_log_submit_to_flush_time=3526, avg_log_submit_to_slide_time=4539) [2025-01-08 00:51:40.513011] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4083, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:40.519676] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=32] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.542263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.544965] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.545796] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:40.545846] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=244, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:40.546282] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:40.559328] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:40.578819] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=75] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:40.601587] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=92, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:40.602278] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.661678] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=45] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:40.661738] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=49] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:40.661758] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=16] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=99) [2025-01-08 00:51:40.680551] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D2248464C-0-0] [lt=97] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:51:40.702448] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.709729] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EF4-0-0] [lt=24] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:40.735284] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58579][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D9A5-0-0] [lt=57] init point thread id with(&point=0x5572d50e5e00, idx_=1235, point=[thread id=58579, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58579) [2025-01-08 00:51:40.740751] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EF5-0-0] [lt=23] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:40.743261] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] success to update min reserved snapshot(reserved_snapshot=1736324494359805000, duration=1800, snapshot_gc_ts=1736326294359805000) [2025-01-08 00:51:40.743303] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:40.743331] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:40.743344] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:40.743448] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=98] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:40.748792] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=26] update reserved snapshot finished(cost_ts=5532, reserved_snapshot=1736324494359805000) [2025-01-08 00:51:40.748153] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=15] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:51:40.749307] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=1149] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:40.754564] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EF6-0-0] [lt=27] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:40.755061] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:40.755166] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44889, [delay/us]:prepare:14, schedule:92, submit:49, rt:1863, total:2020) [2025-01-08 00:51:40.755254] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=81] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:40.755310] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44887, [delay/us]: prepare:14, schedule:92, submit:49, device:1863, total:2020) [2025-01-08 00:51:40.755374] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:40.755420] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4864, iops:16, bw:77529, [delay/us]:prepare:8, schedule:24, submit:47, rt:1170, total:1249) [2025-01-08 00:51:40.755432] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:40.755471] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4864, avg_iops:16, avg_bw:77532, [delay/us]: prepare:8, schedule:24, submit:47, device:1170, total:1249) [2025-01-08 00:51:40.755483] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:40.755505] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16323, [delay/us]:prepare:6, schedule:67, submit:1, rt:1851, total:1926) [2025-01-08 00:51:40.755527] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:42, bw:171393, [delay/us]:prepare:10, schedule:189, submit:276, rt:985, total:1461) [2025-01-08 00:51:40.755536] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:40.755564] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16323, [delay/us]: prepare:6, schedule:67, submit:1, device:1851, total:1926) [2025-01-08 00:51:40.755580] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:42, avg_bw:171393, [delay/us]: prepare:10, schedule:189, submit:276, device:985, total:1461) [2025-01-08 00:51:40.755590] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:40.755610] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4080, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:269, total:300) [2025-01-08 00:51:40.755634] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:40.755661] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4081, [delay/us]: prepare:0, schedule:0, submit:600000000, device:269, total:300) [2025-01-08 00:51:40.755679] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:40.755693] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:40.755702] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:40.755759] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:40.755775] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SERVER] net_in=240kB/s, net_out=227kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:40.758104] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=58] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:40.758133] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:40.758146] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:40.758153] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:40.758160] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:40.758231] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=68] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=93) [2025-01-08 00:51:40.759735] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=73] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:40.774919] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EF7-0-0] [lt=46] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:40.776816] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=47] failed to get result(ret=-4008) [2025-01-08 00:51:40.776940] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=38] check new jobs(ret=0, tenant_id=1, is_oracle_tenant=false, job_infos=[{tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326178662155, start_date_:1736323478339648, last_date_:1736326178341674, this_date_:0, next_date_:1736327078339648, end_date_:64060560000000000, total_:8578451, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736408978339542, last_date_:0, this_date_:0, next_date_:1736408978339542, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:51:40.777098] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=154] check all tenants(ret=0) [2025-01-08 00:51:40.779447] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:40.779714] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] [CACHE] tenant_id= 1 cache memory info: [CACHE] tenant_id= 1 | cache_name= schema_cache | cache_size= 54098944 | cache_store_size= 52019200 | cache_map_size= 2079744 | kv_cnt= 3298 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tablet_table_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 10 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= vtable_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 54 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1176 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_block_cache | cache_size= 20806656 | cache_store_size= 18726912 | cache_map_size= 2079744 | kv_cnt= 2397 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 22 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= bf_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 3 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= fuse_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 56 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= storage_meta_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 1126 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 8 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= log_kv_cache | cache_size= 89472000 | cache_store_size= 87392256 | cache_map_size= 2079744 | kv_cnt= 1287 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 191 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 275 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_ds_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 31 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-01-08 00:51:40.779834] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=67] [CACHE] tenant_id= 1001 cache memory info: [CACHE] tenant_id= 1001 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 989 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1514 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= storage_meta_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 992 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= log_kv_cache | cache_size= 47856640 | cache_store_size= 45776896 | cache_map_size= 2079744 | kv_cnt= 663 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 58 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 242 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-01-08 00:51:40.779954] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=11] [CACHE] tenant_id= 1002 cache memory info: [CACHE] tenant_id= 1002 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 710 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1066 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 712 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= log_kv_cache | cache_size= 24968192 | cache_store_size= 22888448 | cache_map_size= 2079744 | kv_cnt= 318 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 47 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 234 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-01-08 00:51:40.779982] INFO print_current_status (ob_kvcache_hazard_version.cpp:328) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=14] [KVCACHE-HAZARD] hazard version status info: current_version: 11222 | min_version= 11222 | total_nodes_count: 3 | [KVCACHE-HAZARD] i= 95 | acquire_version=18446744073709551615 | waiting_nodes_count= 3 | last_retire_version= 11219 | [2025-01-08 00:51:40.788253] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EF9-0-0] [lt=12] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:40.790359] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=15] ====== tenant freeze timer task ====== [2025-01-08 00:51:40.792827] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EFA-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:40.803342] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.806117] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=23] [APPLY STAT CB APPEND COST TIME](cur_stat_count=44, stat_interval=5000000, avg (count/cost)=152, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:40.806503] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=383] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=44, stat_interval=5000000, avg (count/cost)=5224, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:40.806518] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=14] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=44, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:40.806526] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=44, stat_interval=5000000, avg (count/cost)=-985, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:40.806533] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=44, stat_interval=5000000, avg (count/cost)=4392, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:40.808397] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3FB-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:40.813031] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:40.813060] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:40.814467] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EFB-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:40.815236] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=54] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:40.815883] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=39] gc stale ls task succ [2025-01-08 00:51:40.816031] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:40.816132] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=94] gc stale ls task succ [2025-01-08 00:51:40.816794] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1}, {id:1002}]) [2025-01-08 00:51:40.818067] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D095-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:40.818718] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87127-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:40.818809] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=65][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:40.818823] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:40.828816] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=28] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:40.835357] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB611-0-0] [lt=30] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:40.835479] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB611-0-0] [lt=100] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326300329087003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326300329087003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:40.841308] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=19] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=649809920, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=423931904, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:40.841395] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=48] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:40.841417] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2863, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:40.841482] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:40.841534] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:40.841547] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:40.852285] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=11] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:40.854877] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=11] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:40.862717] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918543F-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:40.862827] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EFC-0-0] [lt=46] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:40.862938] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:40.862960] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:51:40.862970] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:40.862983] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:40.862996] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193804774743, flush_io_finished_ret_:0, flush_io_finished_round_:1959, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:40.865545] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EFD-0-0] [lt=21] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:40.875577] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=81] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:40.876494] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=17] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:40.876662] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=74] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:40.876744] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:40.879408] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=102] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:40.885317] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:40.890346] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=10] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2219281}, scn={val:1736326300329087005, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:40.890405] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:26, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326300890164, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2219337}, next_to_submit_scn_:{val:1736326300329087006, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2219225}, end_lsn:{lsn:2219337}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2219337}, readable_log_tail:{lsn:2219337}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64885431, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326300329087005, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326300329087005, v:0}, data_checksum:2399972301, flag:27497}}, init_mode_version:0, accumulate_checksum:777863780, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326300329087005, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18143, miss_cnt:0, cache_read_size:2219337, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2219281}, scn_:{val:1736326300329087005, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326300902478, first_handle_ts_:1736326300890403, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad962c04150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:51:40.890971] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=34] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324495661593000) [2025-01-08 00:51:40.891000] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2219281}, scn_:{val:1736326300329087005, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326300902478, first_handle_ts_:1736326300890953, replay_cost_:45, retry_cost_:-1, read_log_buf_:0x2ad962c04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326300890164, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2219337}, next_to_submit_scn_:{val:1736326300329087006, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2219225}, end_lsn:{lsn:2219337}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2219337}, readable_log_tail:{lsn:2219337}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64885431, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326300329087005, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326300329087005, v:0}, data_checksum:2399972301, flag:27497}}, init_mode_version:0, accumulate_checksum:777863780, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326300329087005, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18143, miss_cnt:0, cache_read_size:2219337, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=14, ret=0) [2025-01-08 00:51:40.896612] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.897140] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.897267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=59] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.897572] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:40.898061] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=74] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:40.903572] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:40.912761] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8759, offset=2487525}, end_cursor:ObLogCursor{file_id=1, log_id=8760, offset=2487830}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bd0050, len:2843, log_data_len:305, seq:8759, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:40.913535] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E68-0-0] [lt=29] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:40.913581] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E68-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326300845068000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326300845068000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:40.915999] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E68-0-0] [lt=32] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.817059612274169922e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.519306803001552453e+02) [2025-01-08 00:51:40.918840] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18713058}, readable_log_tail:{lsn:18713058}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48391710, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:40.919041] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=197] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:40.919059] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=16] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2197987}, readable_log_tail:{lsn:2197987}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64906781, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:40.919132] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=71] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2197987}, readable_log_tail:{lsn:2197987}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64906781, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:40.919180] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=47] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:40.919315] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=130] get_replay_process success(id={id:1002}, submitted_log_size=2231746, unsubmitted_log_size=2231746, replayed_log_size=2231746, unreplayed_log_size=0) [2025-01-08 00:51:40.919326] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=7] get_replay_process success(id={id:1003}, submitted_log_size=2219337, unsubmitted_log_size=2219337, replayed_log_size=2219337, unreplayed_log_size=0) [2025-01-08 00:51:40.919354] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=7] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:40.923909] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=65] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8760, offset=2487830}, end_cursor:ObLogCursor{file_id=1, log_id=8761, offset=2488127}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894be0050, len:2538, log_data_len:297, seq:8760, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:40.924450] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E68-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:40.924488] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E68-0-0] [lt=26] get_max_applied_scn(scn={val:1736326300845068000, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2197987}, palf_committed_end_scn_:{val:1736326300845068000, v:0}, last_check_scn_:{val:1736326300845068000, v:0}, max_applied_cb_scn_:{val:1736326300845068000, v:0}}) [2025-01-08 00:51:40.924515] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E68-0-0] [lt=20] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326300845068000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326300845068000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:40.935926] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=143] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18144, miss_cnt:0, cache_read_size:2219459, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:40.938001] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=57] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8761, offset=2488127}, end_cursor:ObLogCursor{file_id=1, log_id=8762, offset=2488424}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bd4050, len:2241, log_data_len:297, seq:8761, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:40.939055] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=52] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8762, offset=2488424}, end_cursor:ObLogCursor{file_id=1, log_id=8763, offset=2488721}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bd8050, len:1944, log_data_len:297, seq:8762, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:40.940544] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C7-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326300957366, keepalive_ts:-1}) [2025-01-08 00:51:40.940590] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C7-0-0] [lt=44] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326300957366, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:40.940661] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58402][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C8-0-0] [lt=39] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326300957495, keepalive_ts:-1}) [2025-01-08 00:51:40.940738] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58402][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2C8-0-0] [lt=76] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326300957495, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:40.941312] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF31-0-0] [lt=66] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326300956940, keepalive_ts:-1}) [2025-01-08 00:51:40.941331] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58549][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF31-0-0] [lt=18] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326300956940, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:40.953900] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=169] PNIO [ratelimit] time: 1736326300953896, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007646, add_bytes: 0 [2025-01-08 00:51:40.958250] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE5-0-0] [lt=26] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:40.958282] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE5-0-0] [lt=25] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=34) [2025-01-08 00:51:40.959917] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:40.977226] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:40.977674] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58569][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=93] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:51:40.980602] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=108] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:40.982487] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=42] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1132452, clean_num=31457) [2025-01-08 00:51:40.982992] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:40.996713] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:40.996839] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=119] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:40.996854] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=12] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:40.996876] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=20] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:41.000124] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=25] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:41.003787] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.004409] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=40] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:41.005896] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=39] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:41.005935] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=37] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:41.007342] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487E-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:41.007425] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487E-0-0] [lt=38] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:41.007442] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487E-0-0] [lt=14] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:41.010156] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=11] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:41.011952] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=98] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:41.014498] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=46] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:41.014538] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=38] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:41.014659] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=33] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:41.014735] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84400151}], scn_array=[cnt:1, {val:1736326300989557000, v:0}], curr_size=585, accum_size=1879, time_cost=12718) [2025-01-08 00:51:41.016309] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487E-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:41.016397] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487E-0-0] [lt=37] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:41.016412] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487E-0-0] [lt=14] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:41.019364] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=12] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:41.021046] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=46] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:41.022712] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=49] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:41.022752] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:41.024403] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487E-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:41.024487] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487E-0-0] [lt=36] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:41.024502] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487E-0-0] [lt=14] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:41.024523] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=11] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:41.025084] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=556] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:41.025156] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=63] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:41.025170] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=12] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:41.025192] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487E-0-0] [lt=20] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:41.025271] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487E-0-0] [lt=77] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:41.025285] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=7] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:41.025292] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:41.025311] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487E-0-0] [lt=16] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:41.025323] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487E-0-0] [lt=11] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:41.025331] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=5] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:41.025336] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=5] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:41.025350] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487E-0-0] [lt=13] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:41.025384] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487E-0-0] [lt=32] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:41.026568] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=9] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:41.026610] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=39] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:41.026622] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=10] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:41.026657] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:41.026668] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=11] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:41.026676] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=7] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:41.026686] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:41.026721] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487E-0-0] [lt=33] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:41.037400] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBE02-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193805951233, cache_obj->added_lc()=false, cache_obj->get_object_id()=13663, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:41.038621] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffadb0, session_id:3221583837, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326285368653, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1496, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:41.038669] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=39] kill and revert session(conn.sessid_=3221583837, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:51:41.038682] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sock ref clean, do destroy(*s={this:0x2ad8efffadb0, session_id:3221583837, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326285368653, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1496, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:41.038795] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=67] unlock session of tenant(conn.sessid_=3221583837, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:51:41.038952] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=39] free session successfully(sessid=3221583837) [2025-01-08 00:51:41.038975] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=23] mark session id unused(sessid=3221583837) [2025-01-08 00:51:41.038810] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221583837, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857AA-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:18, cur_pkt_pos_:24, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:16353211, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:16353212, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:16353213, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:16353213, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:16353213, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:16353213, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:641, com_seq:1}, obp20_header_:{payload_len:613, pkt_seq:1, req_id:16353213, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:13}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:16353213, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:14}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:16353213, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:15}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{receive:obp20_header_:{payload_len:194, pkt_seq:0, req_id:16353214, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:190, rec_:194, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[11]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:16353214, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:16353214, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[13]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:16353214, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[14]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:16353214, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[15]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:16353214, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[16]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:16353214, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:13}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[17]:{receive:obp20_header_:{payload_len:410, pkt_seq:0, req_id:16353215, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:406, rec_:410, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[18]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:16353215, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[19]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:16353215, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[20]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:16353215, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[21]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:16353215, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:1, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[22]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:16353215, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[23]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:16353215, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:10}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:41.050675] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=12] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=154, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:41.050708] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=33] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=4, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:41.056435] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=76, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:41.056511] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:41.056670] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=64, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:41.056693] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:41.056823] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:292K/70:71307 disk:292K/70:71307 t1m2:40K/9:91128 t1m2g0:40K/9:91128 t1002m2:184K/46:74540 t1002m2g0:184K/46:74540 t1001m2:68K/15:49499 t1001m2g0:68K/15:49499 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:41.059115] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:41.071520] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:41.075495] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=67] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326297046573, last_first_register_time_us=1736324449183670, register_time_us=1736326297046573, curr_time_us=1736326301075492) [2025-01-08 00:51:41.075579] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=51] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326301075548, last_submit_register_req_time_us=1736326301075548, register_parent_reason=1) [2025-01-08 00:51:41.075596] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:41.076168] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BB3-0-0] [lt=39] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:41.105332] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.109098] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB3-0-0] [lt=99][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:41.111769] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3538, avg_group_log_size=353) [2025-01-08 00:51:41.114640] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:41.115551] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB3-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:41.116578] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58581][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=24] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:51:41.135619] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A3F-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=16, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:41.135700] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A3F-0-0] [lt=80] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=11981, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:41.135711] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A3F-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=25, avg_log_submit_to_flush_time=1359, avg_log_submit_to_slide_time=11955) [2025-01-08 00:51:41.135725] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A3F-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18235, log_task={header:{begin_lsn:{lsn:2219459}, end_lsn:{lsn:2219581}, log_id:18235, min_scn:{val:1736326300832806001, v:0}, max_scn:{val:1736326300832806001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2219337}, prev_proposal_id:1, committed_end_lsn:{lsn:2219459}, data_checksum:940009244, accum_checksum:164549828, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326301033656, freeze_ts:1736326301033656, submit_ts:1736326301033669, flushed_ts:1736326301034577, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:908}, fs_cb_begin_ts=1736326301135608, log_life_time=101952) [2025-01-08 00:51:41.135813] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:41.144309] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58554][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF63-0-0] [lt=109] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326301259982, keepalive_ts:-1}) [2025-01-08 00:51:41.144341] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58554][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF63-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326301259982, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:41.149615] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FA-0-0] [lt=51] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326301166050, keepalive_ts:-1}) [2025-01-08 00:51:41.149657] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FA-0-0] [lt=40] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326301166050, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:41.156561] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:41.156602] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:41.156720] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:41.156809] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:41.158686] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:41.161105] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1415520, replace_num=15728) [2025-01-08 00:51:41.161134] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:41.181347] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=74] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9387) [2025-01-08 00:51:41.181419] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9387) [2025-01-08 00:51:41.182960] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:41.183773] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3AE8-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=16, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:41.183808] INFO [LIB] stat (utility.h:1210) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3AE8-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=3247, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:41.184482] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326300930114000, v:0}, new_max_version={val:1736326300930114000, v:0}, last_min_version={val:1736326300670851001, v:0}, last_max_version={val:1736326300670851001, v:0}, total_time_us=6058, delta_us=254366) [2025-01-08 00:51:41.184517] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326300930114000, v:0}, min_version={val:1736326300930114000, v:0}, max_version={val:1736326300930114000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:41.193494] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] ====== check clog disk timer task ====== [2025-01-08 00:51:41.193536] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=38] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:41.205662] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.213862] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:41.213994] WDIAG read_multi_row (ob_table_access_helper.h:238) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:41.215606] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B4BBAD7-0-0] [lt=36] get number(ret=0, service_type_=0, range=1, base_id=1736326301213935000, start_id=1736326301213935000, end_id=1736326301213935001) [2025-01-08 00:51:41.216122] INFO delete_row (ob_table_access_helper.h:339) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326301214013) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326301214013) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:41.216251] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:41.216313] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:51:41.216365] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:51:41.218924] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB APPEND COST TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=159, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:41.218990] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=65] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=3915, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:41.219000] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:41.219008] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=-642, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:41.219015] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=27, stat_interval=5000000, avg (count/cost)=3431, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:41.227847] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BB-0-0] [lt=19] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:41.227916] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BB-0-0] [lt=52] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326300843864002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326300843864002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:41.246402] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47A1-0-0] [lt=30] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:41.258462] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1543, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:41.258497] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:41.258517] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=84, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:41.258619] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=89] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:41.259478] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:41.259525] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=78, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:41.259558] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=32] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=4525, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:41.259569] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:41.259576] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=-1212, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:41.259583] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=3391, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:41.272386] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=80, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:41.272646] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:41.273480] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=49] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=1346, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:41.273513] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=124, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:41.273544] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=18, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:41.276389] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58402][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=14] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:41.284707] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=37] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:41.284816] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=59] ====== check clog disk timer task ====== [2025-01-08 00:51:41.284841] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:41.300723] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326300989557000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=311153, in_cluster_service=true, cluster_version={val:1736326300989557000, v:0}, min_cluster_version={val:1736326300989557000, v:0}, max_cluster_version={val:1736326300989557000, v:0}, get_cluster_version_err=0, cluster_version_delta=311153, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36717, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=555797697, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:41.305792] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.319573] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=79] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.319613] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.334257] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2232290}, scn={val:1736326300843864004, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:41.334532] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=38] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324495661593000) [2025-01-08 00:51:41.334560] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2232290}, scn_:{val:1736326300843864004, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326301445832, first_handle_ts_:1736326301334515, replay_cost_:41, retry_cost_:-1, read_log_buf_:0x2ad968204160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326301334032, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2232346}, next_to_submit_scn_:{val:1736326300843864005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2232234}, end_lsn:{lsn:2232346}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2232346}, readable_log_tail:{lsn:2232346}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64872422, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326300843864004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326300843864004, v:0}, data_checksum:1586564255, flag:21573}}, init_mode_version:0, accumulate_checksum:581628939, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326300843864004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18248, miss_cnt:0, cache_read_size:2232346, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=8, ret=0) [2025-01-08 00:51:41.336469] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2223921, this={dio_aligned_buf:{buf_write_offset:4011, buf_padding_size:85, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:24255, truncate_used_ts:5506}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2220032, buf_len=122, total_write_size=12172, total_write_size_after_dio=417792, ob_pwrite_used_ts=228852, count=100, each_pwrite_cost=2288) [2025-01-08 00:51:41.337145] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:41.337262] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=107] temporary file module use no blocks [2025-01-08 00:51:41.337291] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:41.337392] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=97] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:41.337408] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193805250882, flush_io_finished_ret_:0, flush_io_finished_round_:1964, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:41.345667] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=118] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=503, get_gts_with_stc_cnt=29, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=29, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:41.345697] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:41.345721] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=497, get_gts_with_stc_cnt=73, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=73, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:41.360628] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D16FBBE13-0-0] [lt=54] get number(ret=0, service_type_=0, range=1, base_id=1736326301356843000, start_id=1736326301356843000, end_id=1736326301356843001) [2025-01-08 00:51:41.361151] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C7BB612-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326301356843000, start_id=1736326301356843000, end_id=1736326301356843001) [2025-01-08 00:51:41.361355] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C7BB612-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9932, avg_group_log_size=620) [2025-01-08 00:51:41.361400] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C7BB612-0-0] [lt=40] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9975) [2025-01-08 00:51:41.361694] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=102] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:41.363042] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8890-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:41.363100] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8890-0-0] [lt=57] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3484, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:41.363113] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8890-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=105, avg_log_submit_to_flush_time=2494, avg_log_submit_to_slide_time=3449) [2025-01-08 00:51:41.363268] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8891-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:41.363283] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8891-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3619, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:41.363369] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1643, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:41.377334] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D4A-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:41.378424] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:41.380297] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:41.380803] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:41.380910] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:41.381154] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:41.381318] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-01-08 00:51:41.383745] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=45] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:41.383995] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58549][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3AEC-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=91, avg_log_submit_to_flush_time=1692, avg_log_submit_to_slide_time=3354) [2025-01-08 00:51:41.384084] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9932, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:41.385540] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:41.401031] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:41.401066] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=27] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=71124160, tenant_id=1001) [2025-01-08 00:51:41.401089] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:41.404680] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=72] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2593, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:41.404739] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=423, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:41.404797] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:41.405929] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.408401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5990) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=24] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000) [2025-01-08 00:51:41.408452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.408621] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=63] get multi version start(ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.408774] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=95] get multi version start(ls_id={id:1}, tablet_id={id:50380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.408874] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.408986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409104] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409256] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409352] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60429}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409412] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409506] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409566] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409825] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.409967] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=87] get multi version start(ls_id={id:1}, tablet_id={id:101011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410056] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846994}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410150] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410312] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=97] get multi version start(ls_id={id:1}, tablet_id={id:50124}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=77] get multi version start(ls_id={id:1}, tablet_id={id:307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410622] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410806] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=73] get multi version start(ls_id={id:1}, tablet_id={id:60457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410918] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.410975] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101098}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411032] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411145] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411202] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411325] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:60342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411384] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101036}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411497] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411676] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101064}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60129}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411791] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411849] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101080}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.411963] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.412021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.412050] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCB-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:41.412076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.412079] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCB-0-0] [lt=26] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:41.412303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.412385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.412463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.412520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.412685] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=102] get multi version start(ls_id={id:1}, tablet_id={id:50373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.412841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:50437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.413052] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=100] get multi version start(ls_id={id:1}, tablet_id={id:101083}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.413260] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=82] get multi version start(ls_id={id:1}, tablet_id={id:494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.413406] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101062}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.413590] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=126] get multi version start(ls_id={id:1}, tablet_id={id:60263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.413712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.413778] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:129}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.413839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.413899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.413959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101018}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414078] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=83] get multi version start(ls_id={id:1}, tablet_id={id:101034}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414810] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.414999] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101084}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=95] get multi version start(ls_id={id:1}, tablet_id={id:50524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415301] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=89] get multi version start(ls_id={id:1}, tablet_id={id:60233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50219}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415617] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101068}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415879] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.415979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.416073] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60140}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.416134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.416280] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=86] get multi version start(ls_id={id:1}, tablet_id={id:50484}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.416430] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.416527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.416630] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.416726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.416788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.416880] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.417008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.417202] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.417387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=100] get multi version start(ls_id={id:1}, tablet_id={id:60226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.417471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.417556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.417627] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.417725] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:60253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.417791] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101073}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.417876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:60383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.418026] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.418092] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:128}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419110] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419186] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419320] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:101099}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419413] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419488] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101092}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419695] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419900] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.419960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.420073] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.420243] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:60484}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.420338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.420439] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.420534] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101031}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.420626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.420722] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.420886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=82] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846983}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.420983] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421044] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421293] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=98] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846990}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421388] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421486] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421665] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:101103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421808] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.421965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422128] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422269] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422381] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=55] get multi version start(ls_id={id:1}, tablet_id={id:60410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422571] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50140}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422632] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422729] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:101067}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422826] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:429}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.422933] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:60165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423266] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423472] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=87] get multi version start(ls_id={id:1}, tablet_id={id:60309}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423564] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101027}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846986}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.423936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424001] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424134] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846977}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424269] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424612] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424670] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:60488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.424910] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425013] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101082}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425072] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425262] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425536] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425632] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.425991] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.426047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.426308] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=52] get multi version start(ls_id={id:1}, tablet_id={id:50420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.426424] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.426500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:60248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.426602] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.426704] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60502}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.426803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:200009}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.426904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.427005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.427254] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.427376] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:60361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.427488] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60116}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.427553] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.427758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=119] get multi version start(ls_id={id:1}, tablet_id={id:101057}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.427825] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:101001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.427887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.427947] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.428004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.428062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.428120] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.428178] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.428327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=92] get multi version start(ls_id={id:1}, tablet_id={id:60381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.428473] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=87] get multi version start(ls_id={id:1}, tablet_id={id:60380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.428651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.428800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=91] get multi version start(ls_id={id:1}, tablet_id={id:60487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.428995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:101033}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.429775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=718] get multi version start(ls_id={id:1}, tablet_id={id:50479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.429926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430260] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430348] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430449] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:60181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430546] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101066}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430606] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430700] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=90] get multi version start(ls_id={id:1}, tablet_id={id:232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.430959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:320}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431048] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846980}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431109] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431167] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431275] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431339] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431397] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431455] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431514] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431572] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431628] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101030}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101094}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431745] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101020}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431860] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50117}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431918] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101044}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.431972] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:100002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432030] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432146] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432307] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432514] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432575] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50395}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432635] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432694] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:118}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432812] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.432989] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433103] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:445}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433162] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101017}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433221] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433357] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=80] get multi version start(ls_id={id:1}, tablet_id={id:101013}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433420] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433477] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433536] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433594] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433653] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433813] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433884] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60256}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.433942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101010}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434048] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101072}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434114] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434173] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434288] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=60] get multi version start(ls_id={id:1}, tablet_id={id:50431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101088}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434407] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101069}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.434941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435000] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435058] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435117] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435176] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435293] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435485] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50128}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435543] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101060}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435602] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50256}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101100}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435779] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435896] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.435954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436067] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=59] get multi version start(ls_id={id:1}, tablet_id={id:50442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436129] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101022}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436436] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436566] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436630] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436759] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=68] get multi version start(ls_id={id:1}, tablet_id={id:50252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436902] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.436965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437026] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437246] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437429] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437491] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437733] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437914] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.437975] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101035}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438034] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:5}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438293] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:50145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438417] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101058}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438478] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438542] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6A7-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193806352375, cache_obj->added_lc()=false, cache_obj->get_object_id()=13664, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:41.438597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101081}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438658] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101086}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101052}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438840] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.438962] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101019}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.439022] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.439082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.439826] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A45-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:41.439876] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A45-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:51:41.439896] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A45-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:51:41.439143] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101079}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.440886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=1572] get multi version start(ls_id={id:1}, tablet_id={id:50307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.440960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441049] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1736326301441046, bytes: 880376316, bw: 0.189440 MB/s, add_ts: 1006269, add_bytes: 199888 [2025-01-08 00:51:41.441115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441287] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441376] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441496] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:1}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441749] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441848] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.441974] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60320}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442033] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442091] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442308] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:200010}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=87] get multi version start(ls_id={id:1}, tablet_id={id:50104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442550] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442738] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442798] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442919] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.442979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443039] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443099] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443315] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443461] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:200006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443580] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:60472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443646] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443709] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.443951] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444011] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444071] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444289] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444418] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=65] get multi version start(ls_id={id:1}, tablet_id={id:60146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444660] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444816] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444871] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.444969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445287] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:60244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445423] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:60439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445485] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445579] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:155}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50309}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445867] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.445966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:453}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446456] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47A5-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:41.446503] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47A5-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1323, avg_group_log_size=120) [2025-01-08 00:51:41.446540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446606] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446665] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50125}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446756] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846979}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.446976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447071] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447307] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447461] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447526] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447860] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.447955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=96] get multi version start(ls_id={id:1}, tablet_id={id:60375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448440] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=59] get multi version start(ls_id={id:1}, tablet_id={id:50188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101043}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448652] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448768] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448879] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=57] get multi version start(ls_id={id:1}, tablet_id={id:50225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448940] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.448999] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449219] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449288] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449412] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=68] get multi version start(ls_id={id:1}, tablet_id={id:60151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449472] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449531] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449704] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449763] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846988}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.449890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.451562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=1493] get multi version start(ls_id={id:1}, tablet_id={id:60246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.451703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101032}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.451795] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:101089}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.451895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:200008}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452339] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=114] get multi version start(ls_id={id:1}, tablet_id={id:50356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452454] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:124}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452658] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452759] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452823] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.452921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:200005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453181] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=167] get multi version start(ls_id={id:1}, tablet_id={id:50332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:60128}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453652] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453722] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.453976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454266] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454386] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454644] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454796] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454883] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.454979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:140}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455067] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:200001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101028}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455269] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455694] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.455988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50155}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.456076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:101021}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.456137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.456293] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.456390] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:50250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.456508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.456599] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.456766] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.456936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:60196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.457067] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=56] get multi version start(ls_id={id:1}, tablet_id={id:50350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.457229] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.457361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:50498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.457688] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=225] get multi version start(ls_id={id:1}, tablet_id={id:111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.457800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:60416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.457867] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.457932] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458079] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458136] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458245] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458351] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458460] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50116}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:60528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458768] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.458953] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101037}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459112] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459274] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101054}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459453] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459548] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.459567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459586] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.459624] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60395}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459729] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=10] get multi version start(ls_id={id:1}, tablet_id={id:60364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459874] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.459931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.460000] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.460105] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50123}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.460169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101041}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.460288] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.460419] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:51:41.461526] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2997, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:41.461566] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2219947}], scn_array=[cnt:1, {val:1736326301331695001, v:0}], curr_size=122, accum_size=1332, time_cost=21013) [2025-01-08 00:51:41.461592] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:51:41.461611] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=181, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:41.462311] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=16] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:41.462559] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:41.462712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.462792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.462853] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.462910] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101097}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.462967] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.463025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.463082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.463139] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101024}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.463257] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.463321] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.463379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.463435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.463492] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.464371] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=187] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2504, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:41.464400] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2232468}], scn_array=[cnt:1, {val:1736326301342923001, v:0}], curr_size=122, accum_size=1201, time_cost=15925) [2025-01-08 00:51:41.463549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.465637] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=37] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:41.465693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=1008] get multi version start(ls_id={id:1}, tablet_id={id:50190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.465842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.465897] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2461, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:41.465907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.465915] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:41.465944] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=190, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:41.465997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:483}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466058] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466305] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101055}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466369] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466576] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466690] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:123}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466920] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.466977] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467034] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467091] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:454}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467274] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467333] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467391] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467446] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101063}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467504] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467560] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467617] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:3}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467845] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:219}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467902] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.467958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846985}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468122] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=58] get multi version start(ls_id={id:1}, tablet_id={id:141}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468181] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:60399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468360] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101040}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468418] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846978}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468570] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468636] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101016}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468790] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.468940] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469000] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101047}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469114] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469168] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469296] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469384] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469445] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469558] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846993}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469729] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469786] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.469959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101059}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470070] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470239] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:50262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470542] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60219}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470673] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470795] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470916] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60155}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.470976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471035] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471284] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471472] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471591] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471779] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=65] get multi version start(ls_id={id:1}, tablet_id={id:50345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.471976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472229] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472370] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472499] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472697] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472819] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472916] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.472976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473038] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473290] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=72] get multi version start(ls_id={id:1}, tablet_id={id:101046}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473354] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473592] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.473991] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:117}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474055] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474219] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474542] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474748] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474833] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.474929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101045}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475027] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475085] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475273] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:502}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475397] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=53] get multi version start(ls_id={id:1}, tablet_id={id:488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475465] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60453}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475900] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.475961] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.476020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101015}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.476167] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:41.477286] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2198597}], scn_array=[cnt:1, {val:1736326301345373001, v:0}], curr_size=122, accum_size=244, time_cost=15823) [2025-01-08 00:51:41.477503] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=91] ====== tenant freeze timer task ====== [2025-01-08 00:51:41.477685] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.477780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.477879] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101078}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.477943] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.478004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.478171] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.478335] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=104] get multi version start(ls_id={id:1}, tablet_id={id:408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.478437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101049}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.478500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.478727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.478820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.478910] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.478974] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101101}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479277] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479370] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479468] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846995}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479531] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479592] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479652] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.479968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:256}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480035] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480094] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101091}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480381] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101008}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480525] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101039}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480828] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.480928] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101029}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481032] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:60385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481261] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481359] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481459] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481786] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481884] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.481942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482038] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482259] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482348] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482466] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482662] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482725] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482844] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.482961] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.483022] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.483081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.483140] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.483555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.483667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:50422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.483770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50129}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.483838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.483930] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:101095}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.483995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101070}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=94] get multi version start(ls_id={id:1}, tablet_id={id:456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484468] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101075}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484862] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846987}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.484986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.485135] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=87] get multi version start(ls_id={id:1}, tablet_id={id:101077}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.485281] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.485385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.485486] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60118}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.485587] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.485688] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101071}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.485778] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.485878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101093}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.485980] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.486068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.486132] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.486327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.486417] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.486555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101065}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.486776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.487039] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=90] get multi version start(ls_id={id:1}, tablet_id={id:309}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.487282] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=95] get multi version start(ls_id={id:1}, tablet_id={id:438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.487410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:50495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.487517] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.487619] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.487714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.487816] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.487918] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:116}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488167] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=93] get multi version start(ls_id={id:1}, tablet_id={id:100003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488421] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:60435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488518] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488707] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101014}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.488921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.489028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:200011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.489118] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.489267] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=89] get multi version start(ls_id={id:1}, tablet_id={id:334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.489367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.489507] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.489643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.489710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.489809] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.489878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:101102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490120] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490241] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:101004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490412] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490473] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490533] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490653] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.490927] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491026] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491269] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491332] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491483] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491653] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491871] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.491996] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50445}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.492158] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:50503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.492388] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.492523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:200007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.492656] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:60463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.492723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846981}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.492823] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.492899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101050}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.492963] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493023] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493140] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493326] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493503] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493659] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60124}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50429}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493833] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50453}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.493997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:101056}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.494050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.494103] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=10] get multi version start(ls_id={id:1}, tablet_id={id:386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.494155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.494207] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.495577] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.495737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.495845] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=53] get multi version start(ls_id={id:1}, tablet_id={id:50244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.495942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496033] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496377] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=133] get multi version start(ls_id={id:1}, tablet_id={id:60207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496468] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50154}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496574] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101023}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496676] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496827] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496914] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101048}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.496976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497064] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60445}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497160] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101026}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497322] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=83] get multi version start(ls_id={id:1}, tablet_id={id:50372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50141}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497683] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:60189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.497997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.498096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.498206] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.498345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=84] get multi version start(ls_id={id:1}, tablet_id={id:523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.498444] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.498541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.498638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.498747] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=52] get multi version start(ls_id={id:1}, tablet_id={id:428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.498847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.498911] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499216] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:101025}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499286] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499474] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60141}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499529] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499622] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499748] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:4}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.499934] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:100005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500222] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500368] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=80] get multi version start(ls_id={id:1}, tablet_id={id:522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846996}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500885] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.500950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501278] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846991}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501339] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:200002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501436] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:200004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501692] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501888] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.501985] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60117}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:101085}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502275] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50454}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502810] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=86] get multi version start(ls_id={id:1}, tablet_id={id:50335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.502960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503078] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503256] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503321] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101087}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60123}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503473] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:50153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503534] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503592] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50483}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503768] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503828] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.503945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.504005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.504063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:484}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.504122] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.504272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.504371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.504433] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.504487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:101051}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.504583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846982}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.505549] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43464118}], scn_array=[cnt:1, {val:1736326301469783000, v:0}], curr_size=588, accum_size=3061, time_cost=28453) [2025-01-08 00:51:41.506266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=114][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.506468] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=91] get multi version start(ls_id={id:1}, tablet_id={id:100006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.506550] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:50473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.506616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.506678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846984}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.506738] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.506799] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.506890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.506952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507106] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507168] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101061}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507298] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507448] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=92] get multi version start(ls_id={id:1}, tablet_id={id:50181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507511] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101076}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507873] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.507936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508034] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60454}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508100] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508277] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508434] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508669] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=86] get multi version start(ls_id={id:1}, tablet_id={id:250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.508943] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509094] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509263] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:154}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509460] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:50388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509749] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846989}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101074}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.509965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.510062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.510150] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.510258] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.510355] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.510452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.510549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101053}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.510647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=42] REACH SYSLOG RATE LIMIT [frequency] [2025-01-08 00:51:41.512754] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3961) [2025-01-08 00:51:41.512782] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:41.515510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=1] get multi version start(ls_id={id:1}, tablet_id={id:101110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324495766421000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324485742394000, max_merged_snapshot=1) [2025-01-08 00:51:41.515668] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CC1-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3961) [2025-01-08 00:51:41.515974] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:116) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8577E-0-0] [lt=1] SSTableGCTask(cost_ts=107993) [2025-01-08 00:51:41.517769] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3961) [2025-01-08 00:51:41.517938] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:41.517960] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5069, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:41.517969] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=134, avg_log_submit_to_flush_time=3512, avg_log_submit_to_slide_time=4935) [2025-01-08 00:51:41.518064] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3961, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:41.519562] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=42] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.541554] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.544027] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.546587] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47A7-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=34, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:41.546618] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47A7-0-0] [lt=30] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20404, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:41.546757] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47A7-0-0] [lt=136] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=50, avg_log_submit_to_flush_time=2960, avg_log_submit_to_slide_time=20353) [2025-01-08 00:51:41.546733] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=99] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1323, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:41.546826] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=84] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18250, miss_cnt:0, cache_read_size:2232590, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:41.546866] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47A7-0-0] [lt=106][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18342, log_task={header:{begin_lsn:{lsn:2232468}, end_lsn:{lsn:2232590}, log_id:18342, min_scn:{val:1736326301342923001, v:0}, max_scn:{val:1736326301342923001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2232346}, prev_proposal_id:1, committed_end_lsn:{lsn:2232468}, data_checksum:1715074492, accum_checksum:2310583995, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326301446163, freeze_ts:1736326301446163, submit_ts:1736326301446482, flushed_ts:1736326301464692, gen_to_freeze cost time:0, gen_to_submit cost time:319, submit_to_flush cost time:18210}, fs_cb_begin_ts=1736326301546563, log_life_time=100400) [2025-01-08 00:51:41.548501] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1323) [2025-01-08 00:51:41.548552] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=366, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:41.548700] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=69, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:41.555404] INFO [SERVER] refresh_avail_memory (ob_table_load_assigned_memory_manager.cpp:117) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] ObTableLoadAssignedMemoryManager::refresh_avail_memory(MTL_ID()=1, avail_memory=107374182, chunk_count_=0, avail_sort_memory_=107374182, avail_memory_=107374182) [2025-01-08 00:51:41.562255] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=213] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:41.568456] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58585][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:41.568521] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:41.584476] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:41.603351] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=72] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=103, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:41.606331] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=68][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.608447] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:41.614325] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BB5-0-0] [lt=32] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:41.672342] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:41.678013] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BED2-0-0] [lt=40] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:41.702958] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AA6-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:41.705136] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5285391965, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:41.705229] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:41.705247] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8bf5f05f0) [2025-01-08 00:51:41.705257] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5285391965, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:41.706424] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.707248] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [50457][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=33] XA scheduler heartbeat task statistics(avg_time=1) [2025-01-08 00:51:41.723333] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=72] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:41.723357] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=23] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:41.723446] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498E-0-0] [lt=43] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:41.723468] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498E-0-0] [lt=20] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:41.756125] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AA7-0-0] [lt=34] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:41.761405] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:41.761476] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44773, [delay/us]:prepare:19, schedule:490, submit:338, rt:2120, total:2968) [2025-01-08 00:51:41.761493] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:41.761607] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=96] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44774, [delay/us]: prepare:19, schedule:490, submit:338, device:2120, total:2968) [2025-01-08 00:51:41.761623] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:41.761706] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:15, bw:69199, [delay/us]:prepare:18, schedule:115, submit:841, rt:2370, total:3345) [2025-01-08 00:51:41.761720] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:41.761760] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:15, avg_bw:69196, [delay/us]: prepare:18, schedule:115, submit:841, device:2370, total:3345) [2025-01-08 00:51:41.761771] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:41.761795] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16281, [delay/us]:prepare:6, schedule:18, submit:1, rt:907, total:933) [2025-01-08 00:51:41.761818] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4291, iops:42, bw:179097, [delay/us]:prepare:6, schedule:180, submit:562, rt:1791, total:2540) [2025-01-08 00:51:41.761827] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:41.761855] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16281, [delay/us]: prepare:6, schedule:18, submit:1, device:907, total:933) [2025-01-08 00:51:41.761870] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4291, avg_iops:42, avg_bw:179097, [delay/us]: prepare:6, schedule:180, submit:562, device:1791, total:2540) [2025-01-08 00:51:41.761880] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:41.761900] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4070, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:494, total:502) [2025-01-08 00:51:41.761924] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:41.761950] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4070, [delay/us]: prepare:0, schedule:0, submit:600000000, device:494, total:502) [2025-01-08 00:51:41.761968] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:41.761983] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:41.761992] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:41.762000] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:41.762009] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=207kB/s, net_out=219kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:41.762708] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:41.780364] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AA8-0-0] [lt=38] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:41.783161] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=37] failed to get result(ret=-4008) [2025-01-08 00:51:41.783329] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=17] check new jobs(ret=0, tenant_id=1002, is_oracle_tenant=false, job_infos=[{tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326232555425, start_date_:1736325329924481, last_date_:1736326229926861, this_date_:0, next_date_:1736327129924481, end_date_:64060560000000000, total_:6030537, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736410829924386, last_date_:0, this_date_:0, next_date_:1736410829924386, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:51:41.783568] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=233] check all tenants(ret=0) [2025-01-08 00:51:41.788160] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:41.803921] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] get number(ret=0, service_type_=0, range=1, base_id=1736326301803380000, start_id=1736326301803380000, end_id=1736326301803380001) [2025-01-08 00:51:41.806650] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.809169] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AAB-0-0] [lt=53] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:41.812400] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=24] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:41.812780] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:41.812804] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:41.813770] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:41.813864] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326294359805000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:41.813891] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=24] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326294359805000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326294359805000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:41.814427] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=112][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84403526}], scn_array=[cnt:1, {val:1736326301803380000, v:0}], curr_size=586, accum_size=1294, time_cost=10175) [2025-01-08 00:51:41.815499] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=11] ====== check clog disk timer task ====== [2025-01-08 00:51:41.815577] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=75] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:41.815595] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=89] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:41.816307] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:51:41.816370] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=76] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:41.816436] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:41.817252] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=42] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1}, {id:1001}, {id:1003}]) [2025-01-08 00:51:41.818901] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC3FE-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:41.819159] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8712D-0-0] [lt=8][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:41.819267] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:41.819283] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:41.821398] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D098-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:41.822463] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AAC-0-0] [lt=105] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:41.833713] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:41.834384] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AAD-0-0] [lt=21] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:41.835245] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=35] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:41.835475] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326295661593000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:41.836822] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AAE-0-0] [lt=22] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:41.837034] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=76] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:41.837071] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=182) [2025-01-08 00:51:41.837261] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50915][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8F8-0-0] [lt=383] get number(ret=0, service_type_=0, range=1, base_id=1736326301834475000, start_id=1736326301834475000, end_id=1736326301834475001) [2025-01-08 00:51:41.837291] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=26] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:41.837368] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=69] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:41.837396] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=26] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=193) [2025-01-08 00:51:41.837691] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58585][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A4D-0-0] [lt=41] init point thread id with(&point=0x5572d50e5f80, idx_=1241, point=[thread id=58585, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58585) [2025-01-08 00:51:41.852473] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=120] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:41.858864] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:41.858929] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=56] temporary file module use no blocks [2025-01-08 00:51:41.858942] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:41.858956] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:41.858971] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193805772566, flush_io_finished_ret_:0, flush_io_finished_round_:3788, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:41.866377] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185440-0-0] [lt=57][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:41.866763] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:41.866789] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-01-08 00:51:41.866800] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:41.866813] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:41.866827] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193805776866, flush_io_finished_ret_:0, flush_io_finished_round_:1960, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:41.872353] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB612-0-0] [lt=125] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:41.872482] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB612-0-0] [lt=108] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326301331695005, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326301331695005, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:41.878586] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=47] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:41.882295] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:41.883547] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=48] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:41.883641] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326295766421000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:41.884943] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:41.885677] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=76] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:41.885757] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:41.886854] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1719CA81-0-0] [lt=52] get number(ret=0, service_type_=0, range=1, base_id=1736326301884728000, start_id=1736326301884728000, end_id=1736326301884728001) [2025-01-08 00:51:41.887994] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=94] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:41.888106] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=125] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:41.891704] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AAF-0-0] [lt=38] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:41.897200] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=101] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.897243] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.897781] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.897834] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:41.898116] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB612-0-0] [lt=5] get number(ret=0, service_type_=0, range=1, base_id=1736326301897260000, start_id=1736326301897260000, end_id=1736326301897260001) [2025-01-08 00:51:41.902776] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=22] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:41.907222] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AB0-0-0] [lt=40] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:41.907538] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:41.940543] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18154, miss_cnt:0, cache_read_size:2220679, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:41.942175] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8763, offset=2488721}, end_cursor:ObLogCursor{file_id=1, log_id=8764, offset=2489026}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bdc050, len:1647, log_data_len:305, seq:8763, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:41.942439] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E69-0-0] [lt=45] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:41.942539] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E69-0-0] [lt=78] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326301845546000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326301845546000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:41.947399] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50497][T1002_L5_G0][T1002][YB42C0A89F8A-00062B2D1BC9C7CA-0-0] [lt=24] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=2.700910444259643555e+02, this=0x2ad8f30a3208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1535, Resource Limit(MB)=1350, Throttle Trigger(MB)=810, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=5.372071216763659507e+00) [2025-01-08 00:51:41.953671] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:491) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E69-0-0] [lt=77] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=0, MAX_GAP=900000000000, new_sync_scn={val:1736326300843864002, v:0}, new_replayable_scn={val:1736326300843864002, v:0}, new_readable_scn={val:1736326300843864002, v:0}, old_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326299840677003, replayable_scn:1736326299840677003, readable_scn:1736326299840677002, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}) [2025-01-08 00:51:41.954511] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:41.961745] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=24] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:41.962724] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=71] PNIO [ratelimit] time: 1736326301962664, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008768, add_bytes: 0 [2025-01-08 00:51:41.963619] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18714701}], scn_array=[cnt:1, {val:1736326301946709000, v:0}], curr_size=545, accum_size=1033, time_cost=15703) [2025-01-08 00:51:41.964473] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:41.965030] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2233078}], scn_array=[cnt:1, {val:1736326301843279001, v:0}], curr_size=122, accum_size=488, time_cost=15345) [2025-01-08 00:51:41.966823] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8764, offset=2489026}, end_cursor:ObLogCursor{file_id=1, log_id=8765, offset=2489323}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894be4050, len:1342, log_data_len:297, seq:8764, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:41.967413] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E69-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:41.967461] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E69-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326301845546000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326301845546000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:41.971110] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] get_min_unreplayed_log_info(lsn={lsn:2233078}, scn={val:1736326301843279001, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326301850922, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2233078}, next_to_submit_scn_:{val:1736326301843279001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2232956}, end_lsn:{lsn:2233078}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2233200}, readable_log_tail:{lsn:2233200}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64871568, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326301843279000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326301843279000, v:0}, data_checksum:3348837161, flag:21719}}, init_mode_version:0, accumulate_checksum:369176768, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326301843279000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18254, miss_cnt:0, cache_read_size:2233078, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:41.976004] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=108] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8765, offset=2489323}, end_cursor:ObLogCursor{file_id=1, log_id=8766, offset=2489620}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894be8050, len:1045, log_data_len:297, seq:8765, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:41.977157] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8766, offset=2489620}, end_cursor:ObLogCursor{file_id=1, log_id=8767, offset=2489917}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bec050, len:748, log_data_len:297, seq:8766, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:41.982780] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:41.985745] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58549][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=19] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:51:41.991784] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=312] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:41.993745] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:41.994789] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58547C-0-0] [lt=23] MediumCheckTask(cost_ts=12) [2025-01-08 00:51:42.007773] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.051012] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58585][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47B1-0-0] [lt=33][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18347, log_task={header:{begin_lsn:{lsn:2233078}, end_lsn:{lsn:2233200}, log_id:18347, min_scn:{val:1736326301843279001, v:0}, max_scn:{val:1736326301843279001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2232956}, prev_proposal_id:1, committed_end_lsn:{lsn:2233078}, data_checksum:1715074492, accum_checksum:877609401, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326301949647, freeze_ts:1736326301949647, submit_ts:1736326301949654, flushed_ts:1736326301965078, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:15424}, fs_cb_begin_ts=1736326302050903, log_life_time=101256) [2025-01-08 00:51:42.052030] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=94] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=146, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:42.052059] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=7, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:42.064487] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=102, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:42.064560] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:42.064617] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=68, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:42.064647] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:42.064970] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=31] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:304K/70:228395 disk:304K/70:228395 t1m2:48K/11:483927 t1m2g0:48K/11:483927 t1002m2:188K/44:211040 t1002m2g0:188K/44:211040 t1001m2:68K/15:91913 t1001m2g0:68K/15:91913 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:42.069546] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88A6-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:42.074608] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=167] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:42.078571] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326298060321, last_first_register_time_us=1736324454612308, register_time_us=1736326298060321, curr_time_us=1736326302078566) [2025-01-08 00:51:42.078693] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=64] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326302078637, last_submit_register_req_time_us=1736326302078637, register_parent_reason=1) [2025-01-08 00:51:42.078727] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:42.080909] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=47] get_max_applied_scn(scn={val:1736326301946709000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18715246}, palf_committed_end_scn_:{val:1736326301946709000, v:0}, last_check_scn_:{val:1736326301946709001, v:0}, max_applied_cb_scn_:{val:1736326301946709000, v:0}}) [2025-01-08 00:51:42.080957] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=41] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18715246}, readable_log_tail:{lsn:18715246}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48389522, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:42.081138] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=148] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:42.081872] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=95][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18715246}], scn_array=[cnt:1, {val:1736326301946709001, v:0}], curr_size=122, accum_size=1155, time_cost=16499) [2025-01-08 00:51:42.081984] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=42] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:42.082009] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=101) [2025-01-08 00:51:42.093293] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=30] LogLoopThread round_cost_time(us)(round_cost_time=19) [2025-01-08 00:51:42.108367] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.115012] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4507, avg_group_log_size=409) [2025-01-08 00:51:42.118672] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46382][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6A9-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193807032502, cache_obj->added_lc()=false, cache_obj->get_object_id()=13665, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:42.127895] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84405243}], scn_array=[cnt:1, {val:1736326301997858001, v:0}], curr_size=122, accum_size=2547, time_cost=12735) [2025-01-08 00:51:42.134918] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB4-0-0] [lt=42][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.136405] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB4-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.138101] INFO [LIB] stat (utility.h:1210) [58538][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A53-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:42.138131] INFO [LIB] stat (utility.h:1210) [58538][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A53-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22155, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:42.138142] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58538][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A53-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=17, avg_log_submit_to_flush_time=3888, avg_log_submit_to_slide_time=22138) [2025-01-08 00:51:42.138287] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:42.141877] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6AE-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326302141024000, start_id=1736326302141024000, end_id=1736326302141024001) [2025-01-08 00:51:42.144697] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=46] new mysql sessid created(conn.sessid_=3221585850, support_ssl=false) [2025-01-08 00:51:42.144848] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221585850, sess.client_addr_="127.0.0.1:51694") [2025-01-08 00:51:42.144886] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] accept one succ(*s={this:0x2ad8a0de8030, session_id:3221585850, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:183}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.144929] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221585851, support_ssl=false) [2025-01-08 00:51:42.144971] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221585851, sess.client_addr_="127.0.0.1:51698") [2025-01-08 00:51:42.144984] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8a0de8db0, session_id:3221585851, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.145009] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221585852, support_ssl=false) [2025-01-08 00:51:42.145040] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221585852, sess.client_addr_="127.0.0.1:51702") [2025-01-08 00:51:42.145051] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0dea030, session_id:3221585852, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:193}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.145105] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221585853, support_ssl=false) [2025-01-08 00:51:42.145144] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221585853, sess.client_addr_="127.0.0.1:51696") [2025-01-08 00:51:42.145157] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8a0deadb0, session_id:3221585853, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:194}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.145441] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221585855, support_ssl=false) [2025-01-08 00:51:42.145489] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221585855, sess.client_addr_="127.0.0.1:51708") [2025-01-08 00:51:42.145503] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x2ad8a0dec030, session_id:3221585855, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:195}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.145628] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221585857, support_ssl=false) [2025-01-08 00:51:42.145674] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221585857, sess.client_addr_="127.0.0.1:51712") [2025-01-08 00:51:42.145687] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x2ad8a0decdb0, session_id:3221585857, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:196}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.145799] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221585858, support_ssl=false) [2025-01-08 00:51:42.145846] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221585858, sess.client_addr_="127.0.0.1:51716") [2025-01-08 00:51:42.145859] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8a0dee030, session_id:3221585858, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:197}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.146439] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.146490] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=24] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.146516] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.146539] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.146558] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.146583] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.146644] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38] new mysql sessid created(conn.sessid_=3221585859, support_ssl=false) [2025-01-08 00:51:42.146694] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221585859, sess.client_addr_="127.0.0.1:51706") [2025-01-08 00:51:42.146709] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x2ad8efffadb0, session_id:3221585859, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:198}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.146713] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7F-0-0] [lt=27] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.146739] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221585854, support_ssl=false) [2025-01-08 00:51:42.146821] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221585854, sess.client_addr_="127.0.0.1:51714") [2025-01-08 00:51:42.146839] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2ad8efff8030, session_id:3221585854, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:200}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.147008] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=177] new mysql sessid created(conn.sessid_=3221585860, support_ssl=false) [2025-01-08 00:51:42.147061] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221585860, sess.client_addr_="127.0.0.1:51700") [2025-01-08 00:51:42.147078] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2ad8efff8db0, session_id:3221585860, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:201}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.147106] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221585861, support_ssl=false) [2025-01-08 00:51:42.147136] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221585861, sess.client_addr_="127.0.0.1:51704") [2025-01-08 00:51:42.147147] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8efffc030, session_id:3221585861, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:202}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.147168] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221585862, support_ssl=false) [2025-01-08 00:51:42.147268] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221585862, sess.client_addr_="127.0.0.1:51710") [2025-01-08 00:51:42.147284] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2ad8efffcdb0, session_id:3221585862, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:203}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.147311] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221585863, support_ssl=false) [2025-01-08 00:51:42.147344] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221585863, sess.client_addr_="127.0.0.1:51718") [2025-01-08 00:51:42.147355] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efe0a030, session_id:3221585863, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:204}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.147915] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CDF-0-0] [lt=38] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12276, add_timestamp:1736326302147899}) [2025-01-08 00:51:42.148676] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285993-0-0] [lt=17] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12276, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:42.148827] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A7F-0-0] [lt=17] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585858, proxy_sessid=0, client_sessid=3221585858, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.148927] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A80-0-0] [lt=31] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.149870] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A80-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585857, proxy_sessid=0, client_sessid=3221585857, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.149953] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A81-0-0] [lt=24] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.150563] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.150655] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.150680] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.150803] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=62] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.151012] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A81-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585853, proxy_sessid=0, client_sessid=3221585853, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.151095] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A82-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.151132] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.151218] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.151244] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.151265] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.151280] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.151297] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.151426] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=112] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.151449] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.152500] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A82-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585854, proxy_sessid=0, client_sessid=3221585854, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.152609] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A83-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.153578] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A83-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585859, proxy_sessid=0, client_sessid=3221585859, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.153683] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A84-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.154433] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A84-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585863, proxy_sessid=0, client_sessid=3221585863, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.154533] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A85-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.154812] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE1-0-0] [lt=73] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11003, add_timestamp:1736326302154804}) [2025-01-08 00:51:42.155588] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A85-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585862, proxy_sessid=0, client_sessid=3221585862, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.155670] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A86-0-0] [lt=24] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.156637] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A86-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585861, proxy_sessid=0, client_sessid=3221585861, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.156719] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A87-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.157442] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:42.157902] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B92-0-0] [lt=92] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11021, add_timestamp:1736326302157898}) [2025-01-08 00:51:42.159855] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE2-0-0] [lt=20][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.160906] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B93-0-0] [lt=19] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12276, add_timestamp:1736326302160899}) [2025-01-08 00:51:42.161716] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B94-0-0] [lt=85][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12276, add_timestamp:1736326302161713}) [2025-01-08 00:51:42.161736] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B94-0-0] [lt=19][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12276, add_timestamp:1736326302161713}) [2025-01-08 00:51:42.161746] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B94-0-0] [lt=8][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=12276, vtable_locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205376, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205376, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287205376, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.165460] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_stat.cpp:301) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B92-0-0] [lt=29][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:42.165644] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:42.167433] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.167598] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=24] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.167626] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.167652] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.167669] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.167686] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.167703] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:42.167729] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:42.167890] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A88-0-0] [lt=56] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.169661] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A88-0-0] [lt=17] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585855, proxy_sessid=0, client_sessid=3221585855, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.169825] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A89-0-0] [lt=68] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.171279] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A89-0-0] [lt=218] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585852, proxy_sessid=0, client_sessid=3221585852, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.171370] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8A-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.172078] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8A-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585851, proxy_sessid=0, client_sessid=3221585851, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.172159] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8B-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:42.172436] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8030, session_id:3221585854, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.172483] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=39] kill and revert session(conn.sessid_=3221585854, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.172538] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffadb0, session_id:3221585859, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.172559] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221585859, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.172567] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sock ref clean, do destroy(*s={this:0x2ad8efff8030, session_id:3221585854, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.172623] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221585854, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.172636] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221585854, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CE2-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:633, rec_:633, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.172676] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36] sock ref clean, do destroy(*s={this:0x2ad8efffadb0, session_id:3221585859, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.172708] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221585859, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.172717] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221585859, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CE1-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:98, rec_:98, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:10}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:11}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.172926] INFO [SERVER] run (obmp_disconnect.cpp:83) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=36] free session successfully(sessid=3221585854) [2025-01-08 00:51:42.172948] INFO [SERVER] run (obmp_disconnect.cpp:90) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=22] mark session id unused(sessid=3221585854) [2025-01-08 00:51:42.172997] INFO [SERVER] run (obmp_disconnect.cpp:83) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221585859) [2025-01-08 00:51:42.173007] INFO [SERVER] run (obmp_disconnect.cpp:90) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221585859) [2025-01-08 00:51:42.173126] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8B-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585850, proxy_sessid=0, client_sessid=3221585850, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.173283] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857BE-0-0] [lt=4] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10008, add_timestamp:1736326302173276}) [2025-01-08 00:51:42.174033] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857BF-0-0] [lt=19] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11017, add_timestamp:1736326302174029}) [2025-01-08 00:51:42.177895] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857BE-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.179885] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857BF-0-0] [lt=24][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:42.179896] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C0-0-0] [lt=22] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:42.180003] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46497][T1_L0_G0][T1001][YB42C0A89F89-00062B2D1B1857C0-0-0] [lt=71] get_stable_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:42.180088] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46497][T1_L0_G0][T1002][YB42C0A89F89-00062B2D1B1857C0-0-0] [lt=11] get_stable_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:42.181986] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:51:42.183651] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285993-0-0] [lt=19] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12276, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302183620, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302183620, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302183620, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.183754] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285993-0-0] [lt=101] success to process renew task(task={tenant_id:1, table_id:12276, add_timestamp:1736326302147899}, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302183620, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302183620, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302183620, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.183803] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285994-0-0] [lt=32] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11003, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:42.186031] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58579][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF64-0-0] [lt=40] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326302261595, keepalive_ts:-1}) [2025-01-08 00:51:42.186055] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58579][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF64-0-0] [lt=24] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326302261595, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:42.185967] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C19C2B6-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1736326302172291000, start_id=1736326302172291000, end_id=1736326302172291001) [2025-01-08 00:51:42.186189] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58545][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC9-0-0] [lt=49] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326302287855, keepalive_ts:-1}) [2025-01-08 00:51:42.186296] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58545][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBC9-0-0] [lt=106] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326302287855, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:42.187297] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2199451}], scn_array=[cnt:1, {val:1736326301946709002, v:0}], curr_size=122, accum_size=1098, time_cost=18805) [2025-01-08 00:51:42.188304] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0deadb0, session_id:3221585853, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.188351] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=36] kill and revert session(conn.sessid_=3221585853, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.188361] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0decdb0, session_id:3221585857, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.188378] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221585857, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.188385] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8a0deadb0, session_id:3221585853, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.188424] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221585853, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.188436] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221585853, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B92-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:688, rec_:688, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:219, seq_:225}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:226}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.188476] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=37] sock ref clean, do destroy(*s={this:0x2ad8a0decdb0, session_id:3221585857, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.188506] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221585857, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.188515] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221585857, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B93-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:368, rec_:368, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.188623] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=25] free session successfully(sessid=3221585853) [2025-01-08 00:51:42.188643] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=19] mark session id unused(sessid=3221585853) [2025-01-08 00:51:42.188748] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221585857) [2025-01-08 00:51:42.188762] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=13] mark session id unused(sessid=3221585857) [2025-01-08 00:51:42.190063] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB1-0-0] [lt=47] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11008, add_timestamp:1736326302190055}) [2025-01-08 00:51:42.190204] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffcdb0, session_id:3221585862, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326302172291, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.190306] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=94] kill and revert session(conn.sessid_=3221585862, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.190318] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffc030, session_id:3221585861, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326302172291, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.190335] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] kill and revert session(conn.sessid_=3221585861, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.190341] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8efffcdb0, session_id:3221585862, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326302172291, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.190501] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221585862, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.190516] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221585862, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857BF-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:549, rec_:549, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:12, seq_:19}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:20}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.190550] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=30] sock ref clean, do destroy(*s={this:0x2ad8efffc030, session_id:3221585861, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326302172291, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.190584] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221585861, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.190593] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221585861, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857BE-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:436, rec_:436, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.190682] INFO [SERVER] run (obmp_disconnect.cpp:83) [46500][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=32] free session successfully(sessid=3221585862) [2025-01-08 00:51:42.190753] INFO [SERVER] run (obmp_disconnect.cpp:90) [46500][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=70] mark session id unused(sessid=3221585862) [2025-01-08 00:51:42.190854] INFO [SERVER] run (obmp_disconnect.cpp:83) [46500][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221585861) [2025-01-08 00:51:42.190868] INFO [SERVER] run (obmp_disconnect.cpp:90) [46500][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=13] mark session id unused(sessid=3221585861) [2025-01-08 00:51:42.191676] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB3-0-0] [lt=5][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:10008, add_timestamp:1736326302191670}) [2025-01-08 00:51:42.191697] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB3-0-0] [lt=21][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:10008, add_timestamp:1736326302191670}) [2025-01-08 00:51:42.191707] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB3-0-0] [lt=9][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=10008, vtable_locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287191166, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287191166, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287191166, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.195940] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB4-0-0] [lt=27][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326302195935}) [2025-01-08 00:51:42.195978] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB4-0-0] [lt=37][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326302195935}) [2025-01-08 00:51:42.195992] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB4-0-0] [lt=13][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=11017, vtable_locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287201742, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287201742, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326287201742, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.196184] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB3-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.200924] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46497][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C0-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.201341] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2233322}], scn_array=[cnt:1, {val:1736326301843279003, v:0}], curr_size=122, accum_size=732, time_cost=12668) [2025-01-08 00:51:42.202266] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6726971392, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:42.202345] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=63][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43470874}], scn_array=[cnt:1, {val:1736326302172291000, v:0}], curr_size=546, accum_size=9775, time_cost=15969) [2025-01-08 00:51:42.202374] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9775) [2025-01-08 00:51:42.202601] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E5F-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9818) [2025-01-08 00:51:42.202744] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=13, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:42.202759] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=8055, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:42.203940] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=57][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dea030, session_id:3221585852, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326302188163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.203985] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=38] kill and revert session(conn.sessid_=3221585852, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.204011] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] sock ref clean, do destroy(*s={this:0x2ad8a0dea030, session_id:3221585852, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326302188163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.204048] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] unlock session of tenant(conn.sessid_=3221585852, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.204060] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221585852, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AB3-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.204104] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0de8030, session_id:3221585850, trace_id:YB42C0A89F89-00062B2D1AE84AB5-0-0, sql_handling_stage:3, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326302202931, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.204123] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221585850, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.204130] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0de8db0, session_id:3221585851, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326302188163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.204145] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221585851, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.204151] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] sock ref clean, do destroy(*s={this:0x2ad8a0de8db0, session_id:3221585851, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326302188163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.204290] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221585851, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.204308] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] connection close(sessid=3221585851, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AB2-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:134, rec_:134, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:7}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.204434] INFO [SERVER] run (obmp_disconnect.cpp:83) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=40] free session successfully(sessid=3221585852) [2025-01-08 00:51:42.204457] INFO [SERVER] run (obmp_disconnect.cpp:90) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=22] mark session id unused(sessid=3221585852) [2025-01-08 00:51:42.204508] INFO [SERVER] run (obmp_disconnect.cpp:83) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221585851) [2025-01-08 00:51:42.204518] INFO [SERVER] run (obmp_disconnect.cpp:90) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221585851) [2025-01-08 00:51:42.204545] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:869) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB5-0-0] [lt=5][errcode=0] server close connection(sessid=3221585850, proxy_sessid=0, stack="0x810107f 0x101960e0 0x10164efb 0x10315ac1 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:42.204558] INFO [SERVER] process (obmp_quit.h:56) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB5-0-0] [lt=11] quit [2025-01-08 00:51:42.204566] WDIAG [RPC.OBMYSQL] async_write_data (ob_sql_sock_session.cpp:163) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB5-0-0] [lt=7][errcode=-4009] sock has error(ret=-4009) [2025-01-08 00:51:42.204573] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1072) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB5-0-0] [lt=7][errcode=-4009] write response fail(ret=-4009) [2025-01-08 00:51:42.204580] WDIAG [SERVER] response (obmp_base.cpp:87) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB5-0-0] [lt=5][errcode=-4009] failed to flush_buffer(ret=-4009) [2025-01-08 00:51:42.204587] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:50) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB5-0-0] [lt=6][errcode=-4009] response rpc result fail(ret=-4009, tmp_ret=-4009) [2025-01-08 00:51:42.204640] WDIAG [SERVER.OMT] process_one (ob_worker_processor.cpp:89) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB5-0-0] [lt=6][errcode=-4009] process request fail(ret=-4009) [2025-01-08 00:51:42.204647] WDIAG [SERVER.OMT] process (ob_worker_processor.cpp:157) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB5-0-0] [lt=6][errcode=-4009] process request fail(ret=-4009) [2025-01-08 00:51:42.204654] WDIAG [SERVER.OMT] process_request (ob_th_worker.cpp:253) [46494][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4009] process request fail(ret=-4009) [2025-01-08 00:51:42.206166] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285994-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11003, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302206142, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302206142, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302206142, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.206420] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285994-0-0] [lt=251] success to process renew task(task={tenant_id:1, table_id:11003, add_timestamp:1736326302154804}, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302206142, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302206142, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302206142, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.206574] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285995-0-0] [lt=41] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11021, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:42.206739] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe0a030, session_id:3221585863, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326302172291, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.206771] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221585863, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.206781] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] sock ref clean, do destroy(*s={this:0x2ad8efe0a030, session_id:3221585863, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326302172291, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.206886] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221585863, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.206900] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221585863, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857C0-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:562, rec_:562, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.207248] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=27] free session successfully(sessid=3221585863) [2025-01-08 00:51:42.207272] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=22] mark session id unused(sessid=3221585863) [2025-01-08 00:51:42.209326] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.211942] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB4-0-0] [lt=35][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:42.212307] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285995-0-0] [lt=22] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11021, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302212285, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302212285, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302212285, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.212350] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285995-0-0] [lt=41] success to process renew task(task={tenant_id:1, table_id:11021, add_timestamp:1736326302157898}, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302212285, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302212285, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302212285, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.212389] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285996-0-0] [lt=27] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12276, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:42.212722] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326301990271000, v:0}, new_max_version={val:1736326301990271000, v:0}, last_min_version={val:1736326301672602000, v:0}, last_max_version={val:1736326301672602000, v:0}, total_time_us=26398, delta_us=222448) [2025-01-08 00:51:42.212750] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326301990271000, v:0}, min_version={val:1736326301990271000, v:0}, max_version={val:1736326301990271000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:42.214486] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:999) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=33] can close safely, do destroy(*s={this:0x2ad8a0de8030, session_id:3221585850, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:183}, err:5, last_decode_time:1736326302202931, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:243, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-01-08 00:51:42.214551] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=41] unlock session of tenant(conn.sessid_=3221585850, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.214621] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221585850, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AB1-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:8, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:133, rec_:133, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:31, seq_:37}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:38}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{receive:obp_mysql_header_{len_:1, rec_:1, seq_:0}, pkt_name:"COM_QUIT", obp_mysql_header_.type_:1, obp_mysql_header_.is_send_:0}], disconnect_state=3) [2025-01-08 00:51:42.214842] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=36] free session successfully(sessid=3221585850) [2025-01-08 00:51:42.214939] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=94] mark session id unused(sessid=3221585850) [2025-01-08 00:51:42.215648] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B94-0-0] [lt=33][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.215828] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B94-0-0] [lt=23][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.216031] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=65][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dec030, session_id:3221585855, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326302188163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.216070] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221585855, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.216133] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] sock ref clean, do destroy(*s={this:0x2ad8a0dec030, session_id:3221585855, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326302188163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.216172] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] unlock session of tenant(conn.sessid_=3221585855, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.216182] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221585855, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AB4-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:241, rec_:241, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.216313] INFO [SERVER] run (obmp_disconnect.cpp:83) [46496][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=33] free session successfully(sessid=3221585855) [2025-01-08 00:51:42.216329] INFO [SERVER] run (obmp_disconnect.cpp:90) [46496][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=15] mark session id unused(sessid=3221585855) [2025-01-08 00:51:42.216965] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A87-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221585860, proxy_sessid=0, client_sessid=3221585860, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:42.217055] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285996-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12276, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302217029, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302217029, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302217029, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.217147] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285996-0-0] [lt=89] success to process renew task(task={tenant_id:1, table_id:12276, add_timestamp:1736326302160899}, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302217029, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302217029, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302217029, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.217261] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285997-0-0] [lt=33] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=10008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:42.220379] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285997-0-0] [lt=17] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10008, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302220337, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302220337, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302220337, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.220467] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285997-0-0] [lt=87] success to process renew task(task={tenant_id:1, table_id:10008, add_timestamp:1736326302173276}, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302220337, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302220337, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302220337, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.220516] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285998-0-0] [lt=32] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11017, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:42.221573] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C1-0-0] [lt=36] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12217, add_timestamp:1736326302221515}) [2025-01-08 00:51:42.221642] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C1-0-0] [lt=43] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12142, add_timestamp:1736326302221639}) [2025-01-08 00:51:42.221733] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C1-0-0] [lt=14] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11035, add_timestamp:1736326302221730}) [2025-01-08 00:51:42.221793] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C1-0-0] [lt=44] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12072, add_timestamp:1736326302221790}) [2025-01-08 00:51:42.221818] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C1-0-0] [lt=17] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12068, add_timestamp:1736326302221817}) [2025-01-08 00:51:42.226764] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285998-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11017, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302226658, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302226658, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302226658, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.226829] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285998-0-0] [lt=138] success to process renew task(task={tenant_id:1, table_id:11017, add_timestamp:1736326302174029}, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302226658, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302226658, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302226658, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.226875] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285999-0-0] [lt=28] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:42.227202] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285999-0-0] [lt=13] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11008, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227188, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227188, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227188, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.227284] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A285999-0-0] [lt=81] success to process renew task(task={tenant_id:1, table_id:11008, add_timestamp:1736326302190055}, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227188, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227188, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227188, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.227326] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599A-0-0] [lt=24] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12217, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227316, sql_renew_time:1736326302227316, is_mark_fail:false}]) [2025-01-08 00:51:42.227341] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599A-0-0] [lt=15] success to process renew task(task={tenant_id:1, table_id:12217, add_timestamp:1736326302221515}, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227316, sql_renew_time:1736326302227316, is_mark_fail:false}]) [2025-01-08 00:51:42.227362] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599B-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12142, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227357, sql_renew_time:1736326302227357, is_mark_fail:false}]) [2025-01-08 00:51:42.227376] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599B-0-0] [lt=13] success to process renew task(task={tenant_id:1, table_id:12142, add_timestamp:1736326302221639}, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227357, sql_renew_time:1736326302227357, is_mark_fail:false}]) [2025-01-08 00:51:42.227418] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599C-0-0] [lt=27] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11035, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227410, sql_renew_time:1736326302227410, is_mark_fail:false}]) [2025-01-08 00:51:42.227441] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599C-0-0] [lt=22] success to process renew task(task={tenant_id:1, table_id:11035, add_timestamp:1736326302221730}, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227410, sql_renew_time:1736326302227410, is_mark_fail:false}]) [2025-01-08 00:51:42.227470] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599D-0-0] [lt=19] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12072, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227465, sql_renew_time:1736326302227465, is_mark_fail:false}]) [2025-01-08 00:51:42.227484] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599D-0-0] [lt=14] success to process renew task(task={tenant_id:1, table_id:12072, add_timestamp:1736326302221790}, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227465, sql_renew_time:1736326302227465, is_mark_fail:false}]) [2025-01-08 00:51:42.227506] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599E-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12068, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227502, sql_renew_time:1736326302227502, is_mark_fail:false}]) [2025-01-08 00:51:42.227520] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599E-0-0] [lt=14] success to process renew task(task={tenant_id:1, table_id:12068, add_timestamp:1736326302221817}, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227502, sql_renew_time:1736326302227502, is_mark_fail:false}]) [2025-01-08 00:51:42.231391] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dee030, session_id:3221585858, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.231460] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=57] kill and revert session(conn.sessid_=3221585858, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.231474] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sock ref clean, do destroy(*s={this:0x2ad8a0dee030, session_id:3221585858, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326302154405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.231523] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221585858, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.231555] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] connection close(sessid=3221585858, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B94-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.231625] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] free session successfully(sessid=3221585858) [2025-01-08 00:51:42.231649] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=22] mark session id unused(sessid=3221585858) [2025-01-08 00:51:42.242468] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=54] ====== traversal_flush timer task ====== [2025-01-08 00:51:42.242639] INFO [MDS] flush (mds_table_mgr.cpp:195) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58547D-0-0] [lt=74] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326301946709002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326310087570000, latest_log_ts:{val:1736326293821063001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.242795] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58547D-0-0] [lt=100] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326297838903000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.242889] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58547D-0-0] [lt=64] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326297339561003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.243361] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58547D-0-0] [lt=113] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326297823944000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.243430] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58547D-0-0] [lt=65] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:42.251155] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47B5-0-0] [lt=33] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:51:42.254821] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2220923}], scn_array=[cnt:1, {val:1736326301834475003, v:0}], curr_size=122, accum_size=976, time_cost=13278) [2025-01-08 00:51:42.259410] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A7F-0-0] [lt=16] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326302259374}) [2025-01-08 00:51:42.259705] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599F-0-0] [lt=11] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:42.268370] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=2188, avg_group_log_size=198) [2025-01-08 00:51:42.268409] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2188) [2025-01-08 00:51:42.268478] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:42.268493] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1342) [2025-01-08 00:51:42.269742] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5682, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:42.269774] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:42.269805] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=87, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:42.269855] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=27, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:42.274147] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get number(ret=0, service_type_=0, range=1, base_id=1736326302264298000, start_id=1736326302264298000, end_id=1736326302264298001) [2025-01-08 00:51:42.274352] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=101] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=83, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:42.274498] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [58586][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=0] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5286193287, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:51:42.275392] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BC-0-0] [lt=56] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:42.275439] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BC-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326301843279003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326301843279003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:42.277407] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2233444}], scn_array=[cnt:1, {val:1736326301843279004, v:0}], curr_size=122, accum_size=854, time_cost=25903) [2025-01-08 00:51:42.277970] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599F-0-0] [lt=26] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302277938, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302277938, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302277938, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.278019] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28599F-0-0] [lt=46] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326302259374}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302277938, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302277938, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302277938, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:42.279058] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:42.281814] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=6405, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:42.281862] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=433, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:42.281905] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=34, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:42.285637] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:42.288859] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] worker thread began to shrink(tenant_->id()=1002, group_id_=20, token=0) [2025-01-08 00:51:42.291495] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=22] ls blacklist refresh finish(cost_time=33177) [2025-01-08 00:51:42.292800] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [58586][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=89] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5286193287, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:51:42.300162] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58309][T1002_L0_G20][T1002][Y0-0000000000000000-0-0] [lt=26] worker thread exit(tenant_->id()=1002, workers_.get_size()=1) [2025-01-08 00:51:42.313884] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.315809] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18715490}], scn_array=[cnt:1, {val:1736326301946709003, v:0}], curr_size=122, accum_size=1399, time_cost=44018) [2025-01-08 00:51:42.317324] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8db0, session_id:3221585860, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326302220305, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.317367] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=37] kill and revert session(conn.sessid_=3221585860, proxy_sessid=0, ret=0) [2025-01-08 00:51:42.317380] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] sock ref clean, do destroy(*s={this:0x2ad8efff8db0, session_id:3221585860, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326302220305, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:42.317415] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221585860, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:42.317426] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221585860, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857C1-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:119, rec_:119, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:2, seq_:4}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:5}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:42.317436] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [46476][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1609BCC5-0-0] [lt=33] handle gts request statistics(total_rt=1838, total_cnt=256, avg_rt=7.151750972762646263e+00, avg_cnt=2.560000000000000142e+01) [2025-01-08 00:51:42.317715] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=8] free session successfully(sessid=3221585860) [2025-01-08 00:51:42.317731] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=15] mark session id unused(sessid=3221585860) [2025-01-08 00:51:42.319694] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.319758] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.335093] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE8F9-0-0] [lt=51] get number(ret=0, service_type_=0, range=1, base_id=1736326302335060000, start_id=1736326302335060000, end_id=1736326302335060001) [2025-01-08 00:51:42.339146] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=36] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:42.339244] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=89] temporary file module use no blocks [2025-01-08 00:51:42.339280] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=34] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:42.339298] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:42.339311] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193806251288, flush_io_finished_ret_:0, flush_io_finished_round_:1965, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:42.339791] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A57-0-0] [lt=37][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18247, log_task={header:{begin_lsn:{lsn:2220923}, end_lsn:{lsn:2221045}, log_id:18247, min_scn:{val:1736326301834475003, v:0}, max_scn:{val:1736326301834475003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2220801}, prev_proposal_id:1, committed_end_lsn:{lsn:2220923}, data_checksum:940009244, accum_checksum:3968423928, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326302237678, freeze_ts:1736326302237679, submit_ts:1736326302237693, flushed_ts:1736326302254965, gen_to_freeze cost time:1, gen_to_submit cost time:15, submit_to_flush cost time:17272}, fs_cb_begin_ts=1736326302339780, log_life_time=102102) [2025-01-08 00:51:42.344487] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1609BCC5-0-0] [lt=122] get number(ret=0, service_type_=0, range=1, base_id=1736326302344378000, start_id=1736326302344378001, end_id=1736326302344378002) [2025-01-08 00:51:42.349702] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ts response task statistics(alloc_count=60626, free_count=30312) [2025-01-08 00:51:42.349735] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:42.365856] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:42.374728] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:42.374871] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=106] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8373, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:42.376655] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88AF-0-0] [lt=261] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:42.376695] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88AF-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14886, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:42.376709] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88AF-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=139, avg_log_submit_to_flush_time=12946, avg_log_submit_to_slide_time=14746) [2025-01-08 00:51:42.376783] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1643, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:42.379707] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:42.381509] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=21] current monitor number(seq_=-1) [2025-01-08 00:51:42.385543] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=78] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:42.385857] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:42.385974] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:42.388320] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=114] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:42.388974] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:42.391269] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=10407, avg_group_log_size=650) [2025-01-08 00:51:42.391300] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10364) [2025-01-08 00:51:42.395336] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58524][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E63-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=947, avg_log_gen_to_submit_time=1047, avg_log_submit_to_flush_time=6880, avg_log_submit_to_slide_time=9282) [2025-01-08 00:51:42.395563] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9818, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:42.403288] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:42.409127] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4395, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:42.409278] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=438, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:42.409324] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:42.414256] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.414271] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCC-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:42.414300] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCC-0-0] [lt=25] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:42.415970] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=78] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:42.429059] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=27] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:42.429940] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=41] ====== traversal_flush timer task ====== [2025-01-08 00:51:42.430117] INFO [MDS] flush (mds_table_mgr.cpp:195) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F2D-0-0] [lt=45] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326302388479000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326309284252000, latest_log_ts:{val:1736326292617740001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.430308] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F2D-0-0] [lt=183] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:42.443803] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1736326302443799, bytes: 880695550, bw: 0.303609 MB/s, add_ts: 1002753, add_bytes: 319234 [2025-01-08 00:51:42.448961] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47B9-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:42.449085] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47B9-0-0] [lt=110] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:42.458351] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.461300] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:51:42.462009] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6AF-0-0] [lt=5][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193807375843, cache_obj->added_lc()=false, cache_obj->get_object_id()=13666, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:42.462350] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.470179] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=28] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:42.470274] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=90] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:42.470288] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:42.470454] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=162] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:42.473771] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] Dumping All Cache Objs(alloc_obj_list.count()=132, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}]) [2025-01-08 00:51:42.474188] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=413] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:42.475315] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=229] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10865, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:42.475342] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:42.475397] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=994, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:42.514353] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.517137] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4528) [2025-01-08 00:51:42.517171] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:42.518547] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4528) [2025-01-08 00:51:42.518728] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4528) [2025-01-08 00:51:42.519781] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.520539] INFO [LIB] stat (utility.h:1210) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6964-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:42.520655] INFO [LIB] stat (utility.h:1210) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6964-0-0] [lt=114] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6475, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:42.520890] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6964-0-0] [lt=230] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=4596, avg_log_submit_to_slide_time=6366) [2025-01-08 00:51:42.521021] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4528, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:42.538555] INFO [LIB] stat (utility.h:1210) [58579][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A5B-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:42.538622] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58579][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A5B-0-0] [lt=56] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:42.538641] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58579][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A5B-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:42.538744] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58579][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A5B-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18249, log_task={header:{begin_lsn:{lsn:2221167}, end_lsn:{lsn:2221289}, log_id:18249, min_scn:{val:1736326302433684000, v:0}, max_scn:{val:1736326302433684000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2221045}, prev_proposal_id:1, committed_end_lsn:{lsn:2221167}, data_checksum:940009244, accum_checksum:3702399488, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326302438474, freeze_ts:1736326302438474, submit_ts:1736326302438489, flushed_ts:1736326302440533, gen_to_freeze cost time:0, gen_to_submit cost time:15, submit_to_flush cost time:2044}, fs_cb_begin_ts=1736326302538656, log_life_time=100182) [2025-01-08 00:51:42.540412] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=124] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2790, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:42.540442] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:42.540485] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=626, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:42.540584] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=66, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:42.542562] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.543906] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=93] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.550494] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=57] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6847, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:42.550624] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=127] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:42.550645] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=483, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:42.550717] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=56, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:42.552282] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88B3-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=31, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:42.552321] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88B3-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=27380, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:42.552302] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=61] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:42.552371] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88B3-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=25, avg_log_submit_to_flush_time=7603, avg_log_submit_to_slide_time=27354) [2025-01-08 00:51:42.552365] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=55] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18261, miss_cnt:0, cache_read_size:2233932, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:42.566053] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:42.566762] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:42.569670] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58579][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=36] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:42.571352] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:42.591144] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8ADF8-0-0] [lt=14] need check schema(new_schema_version=1736326191428712, cached_tenant_schema_version=1736326191428712, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=update __all_weak_read_service set min_version=?, max_version=? where tenant_id = ? and level_id = ? and level_value = ? and min_version = ? and max_version = ?) [2025-01-08 00:51:42.603794] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:42.607814] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=80, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:42.607880] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] submit log success(service_type_=0, last_id=1736326319254996000, limited_id=1736326319254996000) [2025-01-08 00:51:42.607896] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] submit log with lock success(service_type_=0, last_id=1736326319254996000, limited_id=1736326319254996000) [2025-01-08 00:51:42.611533] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=13] update id meta(ret=0, service_type=0, limited_id=1736326319254996000, latest_scn={val:1736326302607470001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319254996000, latest_log_ts:{val:1736326302607470001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:42.611598] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=65] handle submit callback(service_type_=0, success=true, used_ts=3719, log_ts={val:1736326302607470001, v:0}, limited_id=1736326319254996000) [2025-01-08 00:51:42.611612] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=12] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:42.611621] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:42.612820] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326302611968000, start_id=1736326302611968000, end_id=1736326302611968001) [2025-01-08 00:51:42.614517] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.669005] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=20] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:42.669042] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] try gc retain ctx [2025-01-08 00:51:42.669050] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:42.669086] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326302349610004, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326292316531004, v:0}, keep_alive_scn_:{val:1736326297338357003, v:0}, update_ts_:1736326302668823}) [2025-01-08 00:51:42.670012] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326310087570000, latest_log_ts:{val:1736326293821063001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:42.670320] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=89] finish update min start scn(max_decided_scn={val:1736326302349610003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326292316531004, v:0}, keep_alive_scn_:{val:1736326297338357003, v:0}, update_ts_:1736326302668823}) [2025-01-08 00:51:42.671045] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=30] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326297838903000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:42.671217] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=61] finish update min start scn(max_decided_scn={val:1736326302434797002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326295328613004, v:0}, keep_alive_scn_:{val:1736326300341303005, v:0}, update_ts_:1736326302668823}) [2025-01-08 00:51:42.671961] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=82] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326297339561003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:42.672027] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=58] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:42.672046] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] get_min_unreplayed_log_info(lsn={lsn:2221533}, scn={val:1736326302433684003, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326302641386, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2221533}, next_to_submit_scn_:{val:1736326302433684003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2221411}, end_lsn:{lsn:2221533}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2221533}, readable_log_tail:{lsn:2221533}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64883235, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326302433684002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326302433684002, v:0}, data_checksum:2124447694, flag:31426}}, init_mode_version:0, accumulate_checksum:1211965737, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326302433684002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18161, miss_cnt:0, cache_read_size:2221533, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:42.672120] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=72] finish update min start scn(max_decided_scn={val:1736326302433684002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326294317339001, v:0}, keep_alive_scn_:{val:1736326299327803001, v:0}, update_ts_:1736326302668823}) [2025-01-08 00:51:42.672910] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326297823944000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:42.676135] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] submit log success(service_type_=0, last_id=1736326319284252000, limited_id=1736326319284252000) [2025-01-08 00:51:42.676157] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] submit log with lock success(service_type_=0, last_id=1736326319284252000, limited_id=1736326319284252000) [2025-01-08 00:51:42.679033] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=36] update id meta(ret=0, service_type=0, limited_id=1736326319284252000, latest_scn={val:1736326302668823001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319284252000, latest_log_ts:{val:1736326302668823001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:42.679359] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=323] handle submit callback(service_type_=0, success=true, used_ts=3225, log_ts={val:1736326302668823001, v:0}, limited_id=1736326319284252000) [2025-01-08 00:51:42.679376] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=16] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:42.679385] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:42.681058] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326302680777000, start_id=1736326302680777000, end_id=1736326302680777001) [2025-01-08 00:51:42.701435] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=33] ====== tenant freeze timer task ====== [2025-01-08 00:51:42.714073] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5287279577, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:42.714170] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:42.714248] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=76] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad9085470f0) [2025-01-08 00:51:42.714258] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5287279577, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:42.714586] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:42.714668] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.766257] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:42.769131] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:42.769267] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=98] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:52836, [delay/us]:prepare:23, schedule:91, submit:313, rt:2444, total:2872) [2025-01-08 00:51:42.769358] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=83] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:42.769447] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=71] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52833, [delay/us]: prepare:23, schedule:91, submit:313, device:2444, total:2872) [2025-01-08 00:51:42.769464] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:42.769504] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4864, iops:16, bw:77218, [delay/us]:prepare:11, schedule:132, submit:677, rt:3578, total:4400) [2025-01-08 00:51:42.769513] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:42.769550] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4864, avg_iops:16, avg_bw:77222, [delay/us]: prepare:11, schedule:132, submit:677, device:3578, total:4400) [2025-01-08 00:51:42.769560] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:42.769588] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16257, [delay/us]:prepare:18, schedule:69, submit:0, rt:906, total:995) [2025-01-08 00:51:42.769632] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:170701, [delay/us]:prepare:8, schedule:391, submit:1158, rt:4735, total:6293) [2025-01-08 00:51:42.769644] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:42.769674] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16256, [delay/us]: prepare:18, schedule:69, submit:0, device:906, total:995) [2025-01-08 00:51:42.769689] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:170697, [delay/us]: prepare:8, schedule:391, submit:1158, device:4735, total:6293) [2025-01-08 00:51:42.769703] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:42.769728] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4064, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:513, total:526) [2025-01-08 00:51:42.769752] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:42.769806] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4064, [delay/us]: prepare:0, schedule:0, submit:600000000, device:513, total:526) [2025-01-08 00:51:42.769828] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:42.769843] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:42.769852] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:42.769860] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:42.769867] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=299kB/s, net_out=259kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:42.777052] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708516B-0-0] [lt=13] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:42.805335] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=98] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:42.811898] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:42.813111] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:42.813143] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:42.814886] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.815487] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC401-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.815958] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:42.816631] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326292807123, end_time=1736326302816628, dag_status={}) [2025-01-08 00:51:42.816660] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:42.816672] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:42.816680] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:42.816687] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:42.816694] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:42.816701] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:42.816708] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:42.816715] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:42.816721] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:42.816729] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816740] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816748] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816756] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816764] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816773] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816780] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816788] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816796] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816805] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816813] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816821] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816829] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816837] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816845] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816853] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816860] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816868] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816877] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816885] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816893] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816901] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816909] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816916] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816924] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816932] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816940] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816948] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816956] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816964] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816972] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816980] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816987] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.816995] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817003] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817011] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817019] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817026] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817034] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817042] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817051] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817059] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817067] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817075] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817083] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817090] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817098] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817106] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817113] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817121] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817129] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817137] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817145] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817153] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817160] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817168] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817176] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817244] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=67] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817253] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817261] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817269] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817277] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817285] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817333] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=47] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817344] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817352] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817360] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817368] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817376] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817384] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817392] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817400] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817408] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:42.817417] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:42.817425] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:42.817432] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:42.817437] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:42.817504] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=66] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:42.817511] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:42.817517] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:42.817522] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:42.817527] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:42.817511] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1}, {id:1003}]) [2025-01-08 00:51:42.817532] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:42.817539] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:42.817832] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=34] gc stale ls task succ [2025-01-08 00:51:42.817982] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=40] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:42.818017] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=12] gc stale ls task succ [2025-01-08 00:51:42.820924] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87133-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.820965] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:42.820977] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:42.822422] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D09B-0-0] [lt=108][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.825672] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84648-0-0] [lt=29] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2025-01-08 00:51:42.827014] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84648-0-0] [lt=22] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2025-01-08 00:51:42.827033] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1010) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84648-0-0] [lt=17] no transfer partition job [2025-01-08 00:51:42.827045] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1040) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84648-0-0] [lt=7] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer=true) [2025-01-08 00:51:42.829952] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:284) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84648-0-0] [lt=10] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-01-08 00:51:42.829987] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:493) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84648-0-0] [lt=23] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2025-01-08 00:51:42.830033] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:157) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84648-0-0] [lt=40] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=3, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1003}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone3", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1736324440080711, last_statistic_bg_stat_time=1736326192603087, last_statistic_schema_version=1736326192037600, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-01-08 00:51:42.835930] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE8FE-0-0] [lt=83] get number(ret=0, service_type_=0, range=1, base_id=1736326302835077000, start_id=1736326302835077000, end_id=1736326302835077001) [2025-01-08 00:51:42.837792] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=17] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:42.837847] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=43] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=89) [2025-01-08 00:51:42.848937] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=97] success to update min reserved snapshot(reserved_snapshot=1736324495766421000, duration=1800, snapshot_gc_ts=1736326295766421000) [2025-01-08 00:51:42.850541] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326302850507], gts:1736326302850005000, latest_srr:[mts=1736326302850507]}) [2025-01-08 00:51:42.850744] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=96] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326302850639], gts:1736326302850005000, latest_srr:[mts=1736326302850639]}) [2025-01-08 00:51:42.851356] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326302851171], gts:1736326302850005000, latest_srr:[mts=1736326302851171]}) [2025-01-08 00:51:42.851596] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853AA-0-0] [lt=11] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:42.851625] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853AA-0-0] [lt=20] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:42.852690] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=34] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:42.853504] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=49] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324495766421000) [2025-01-08 00:51:42.853632] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=124] update reserved snapshot finished(cost_ts=4700, reserved_snapshot=1736324495766421000) [2025-01-08 00:51:42.859366] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:42.859419] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=44] temporary file module use no blocks [2025-01-08 00:51:42.859432] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:42.859447] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:42.859460] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193806772845, flush_io_finished_ret_:0, flush_io_finished_round_:3789, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:42.868710] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185441-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:42.868916] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:42.868935] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-01-08 00:51:42.868942] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:42.868953] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:42.868964] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193806780699, flush_io_finished_ret_:0, flush_io_finished_round_:1961, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:42.879888] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=96] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:42.880725] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] ====== traversal_flush timer task ====== [2025-01-08 00:51:42.880856] INFO [MDS] flush (mds_table_mgr.cpp:195) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857D8-0-0] [lt=24] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326302801436000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319254996000, latest_log_ts:{val:1736326302607470001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.880959] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857D8-0-0] [lt=97] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:42.885814] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:42.885990] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:42.886290] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:42.889294] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-01-08 00:51:42.892084] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C2-0-0] [lt=48][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193807805916, cache_obj->added_lc()=false, cache_obj->get_object_id()=13667, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:42.893101] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C3-0-0] [lt=48][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:42.895149] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C5-0-0] [lt=24][errcode=-4201] hashtable not init [2025-01-08 00:51:42.896766] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.896803] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.900716] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=32] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:42.901081] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.901108] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:42.904940] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:42.913886] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB613-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:42.914003] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB613-0-0] [lt=53] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326302835077000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326302835077000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:42.915049] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:42.931807] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=39] ====== checkpoint timer task ====== [2025-01-08 00:51:42.932759] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:42.933131] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=371] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:42.933310] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326310087570000, latest_log_ts:{val:1736326293821063001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.933484] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=169] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:42.933506] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:42.933521] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1736326293821063001, v:0}) [2025-01-08 00:51:42.933530] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:42.933545] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:42.933564] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:42.933577] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:42.933595] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:42.933616] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:42.934457] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:42.934490] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=33] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:42.934531] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326297838903000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.934590] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=56] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:42.934601] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:42.934613] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:42.934627] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:42.934642] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2200418}, scn={val:1736326302850005000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:42.935639] INFO [PALF] pread (log_reader.cpp:130) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=939) [2025-01-08 00:51:42.936329] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=31] BlockGCTimerTask success(ret=0, cost_time_us=14, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:42.937377] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=76] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8767, offset=2489917}, end_cursor:ObLogCursor{file_id=1, log_id=8768, offset=2490214}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bf0050, len:451, log_data_len:297, seq:8767, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:42.937436] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=43] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:42.937808] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326302850005000, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:42.937842] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:42.937863] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:42.939008] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:42.939043] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=35] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:42.939084] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326297339561003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.939284] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=195] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:42.939303] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:42.939318] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:42.939332] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:42.939347] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2234298}, scn={val:1736326302846602000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:42.941688] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8768, offset=2490214}, end_cursor:ObLogCursor{file_id=1, log_id=8769, offset=2490511}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bf8050, len:4250, log_data_len:297, seq:8768, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:42.941752] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326302846602000, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:42.941842] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:42.941864] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:42.942768] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=201] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:42.942802] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=34] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:42.942843] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326297823944000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326299840677003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:42.942906] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=61] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:42.942918] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:42.942933] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:42.942946] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:42.942959] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2221777}, scn={val:1736326302835077000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:42.944002] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18164, miss_cnt:0, cache_read_size:2221899, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:42.945506] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8769, offset=2490511}, end_cursor:ObLogCursor{file_id=1, log_id=8770, offset=2490808}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bf4050, len:3953, log_data_len:297, seq:8769, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:42.945923] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326302835077000, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:42.945954] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:42.945973] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:42.945982] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:42.961822] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE6-0-0] [lt=24] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:42.961861] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE6-0-0] [lt=31] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=41) [2025-01-08 00:51:42.966476] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:42.969102] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=101] PNIO [ratelimit] time: 1736326302969099, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006435, add_bytes: 0 [2025-01-08 00:51:42.992083] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:42.992248] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58524][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=28] worker thread exit(tenant_->id()=1001, workers_.get_size()=3) [2025-01-08 00:51:42.992499] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1001, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18079, max_lsn:2200540, max_scn:{val:1736326302850005001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2200540}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2200418}, last_submit_end_lsn:{lsn:2200540}, last_submit_log_id:18079, last_submit_log_pid:1, max_flushed_lsn:{lsn:2200418}, max_flushed_end_lsn:{lsn:2200540}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2200540}, last_slide_log_id:18079, last_slide_scn:{val:1736326302850005001, v:0}, last_slide_lsn:{lsn:2200418}, last_slide_end_lsn:{lsn:2200540}, last_slide_log_pid:1, last_slide_log_accum_checksum:3125587289, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924f6a530}, [StateMgr]={this:0x2ad924f6e830, self:"192.168.159.137:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18080, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324447115984, palf_role_change_cb:0x2ad924f752f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924f6bc70}, [ModeMgr]={palf_id:1001, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324447762546, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436756804001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/meta", dir_fd:177, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924f70a38, flashback_version:0}, log_storage:{log_tail:{lsn:2200540}, readable_log_tail:{lsn:2200540}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64904228, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1736324447114985, this:0x2ad924f6ef70}, [Reconfirm]={palf_id:1001, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:1"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324447117136, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924f6e9b0}) [2025-01-08 00:51:42.992815] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=310] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326302972025, last_flushed_end_lsn:{lsn:2200540}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326302973135, last_flushed_end_lsn:{lsn:2200540}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326302975526, last_flushed_end_lsn:{lsn:2200540}}]) [2025-01-08 00:51:42.992884] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=63] current log_task status(palf_id=1001, self="192.168.159.137:2882", log_id=18080, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326302972025, last_flushed_end_lsn:{lsn:2200540}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326302973135, last_flushed_end_lsn:{lsn:2200540}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326302975526, last_flushed_end_lsn:{lsn:2200540}}]) [2025-01-08 00:51:42.994057] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=97] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8770, offset=2490808}, end_cursor:ObLogCursor{file_id=1, log_id=8771, offset=2491113}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894bfc050, len:3656, log_data_len:305, seq:8770, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:42.997085] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6A-0-0] [lt=35] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:42.997158] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6A-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326302850005001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326302850005001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:43.005770] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:43.007908] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=131] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1163909, clean_num=31457) [2025-01-08 00:51:43.009858] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690FC0-0-0] [lt=36] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.787115077972412109e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=893, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.792130180696641517e+01) [2025-01-08 00:51:43.011811] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=78] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8771, offset=2491113}, end_cursor:ObLogCursor{file_id=1, log_id=8772, offset=2491410}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c08050, len:3351, log_data_len:297, seq:8771, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:43.012522] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6A-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:43.012622] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6A-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326302850005001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326302850005001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:43.016151] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.023751] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=44] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8772, offset=2491410}, end_cursor:ObLogCursor{file_id=1, log_id=8773, offset=2491707}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c04050, len:3054, log_data_len:297, seq:8772, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:43.025686] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=44] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8773, offset=2491707}, end_cursor:ObLogCursor{file_id=1, log_id=8774, offset=2492004}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c0c050, len:2757, log_data_len:297, seq:8773, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:43.045124] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=20] runTimer to refresh locality_info(ret=0) [2025-01-08 00:51:43.047749] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:346) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=20] set_server_locality_array success(server_locality_array=[{inited:true, addr:"192.168.159.137:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322580423131, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.138:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581270010, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.139:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581175973, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-01-08 00:51:43.047824] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=60] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-01-08 00:51:43.047839] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=14] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-01-08 00:51:43.047855] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=10] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1001, primary_zone=RANDOM) [2025-01-08 00:51:43.047862] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=7] add_locality_zone success(locality_zone={tenant_id:1001, region_priority:18446744073709551615}) [2025-01-08 00:51:43.047875] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1002, primary_zone=RANDOM) [2025-01-08 00:51:43.047882] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=7] add_locality_zone success(locality_zone={tenant_id:1002, region_priority:18446744073709551615}) [2025-01-08 00:51:43.047851] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185132-0-0] [lt=35] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326298379519, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:43.047891] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1001, region_priority:18446744073709551615}, {tenant_id:1002, region_priority:18446744073709551615}]}) [2025-01-08 00:51:43.047897] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185132-0-0] [lt=44] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326298381311, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:43.047919] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185132-0-0] [lt=21] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, renew_time:1736326298383017, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, renew_time:1736326298383019, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, renew_time:1736326298383021, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326298383014, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:43.048416] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:801) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F852A0-0-0] [lt=25] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, location:{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, location:{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}]) [2025-01-08 00:51:43.053318] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=95, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:43.053388] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=69] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=8, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:43.070834] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=97] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=100, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:43.070891] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:43.071016] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:43.071053] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:43.071652] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:336K/78:298915 disk:336K/78:298915 t1m2:52K/12:44953 t1m2g0:52K/12:44953 t1002m2:208K/50:342892 t1002m2g0:208K/50:342892 t1001m2:76K/16:351959 t1001m2g0:76K/16:351959 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:43.074531] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A94F8-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:43.076782] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:43.116156] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6B1-0-0] [lt=356][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193808029987, cache_obj->added_lc()=false, cache_obj->get_object_id()=13668, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:43.116752] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.116881] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:43.118102] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=240] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=4223, avg_group_log_size=351) [2025-01-08 00:51:43.144923] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A94F9-0-0] [lt=90] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:43.144990] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A94F9-0-0] [lt=65] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21190, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:43.145004] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A94F9-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=18, avg_log_submit_to_flush_time=3176, avg_log_submit_to_slide_time=21172) [2025-01-08 00:51:43.150642] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB5-0-0] [lt=58][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:43.152068] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB5-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:43.152348] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=177] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:43.167656] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:43.173241] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A94FA-0-0] [lt=60] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:51:43.193217] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=114] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1736326302994608000, v:0}, base_version={val:1736326302791572000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326302994608000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326303193198, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326303018599000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326303282064, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326303018599000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326303169749, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:43.193685] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=100] ====== check clog disk timer task ====== [2025-01-08 00:51:43.193711] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:43.195098] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=97] get number(ret=0, service_type_=0, range=1, base_id=1736326303192030000, start_id=1736326303192030000, end_id=1736326303192030001) [2025-01-08 00:51:43.208525] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:43.213223] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326303004177000, v:0}, new_max_version={val:1736326303004177000, v:0}, last_min_version={val:1736326302801436000, v:0}, last_max_version={val:1736326302801436000, v:0}, total_time_us=5052, delta_us=209043) [2025-01-08 00:51:43.213258] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326303004177000, v:0}, min_version={val:1736326303004177000, v:0}, max_version={val:1736326303004177000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:43.217598] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.234632] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F852A1-0-0] [lt=74] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:43.250695] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58554][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47C9-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:43.271147] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:43.271255] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=101] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:43.271397] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:43.271415] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:43.272982] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3477, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:43.273010] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:43.273024] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=315, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:43.273123] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:43.277251] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=68, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:43.278875] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10578) [2025-01-08 00:51:43.279079] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=33] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10535) [2025-01-08 00:51:43.279476] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B08-0-0] [lt=23] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=16, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:43.279511] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B08-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=5783, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:43.285083] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] ====== check clog disk timer task ====== [2025-01-08 00:51:43.285115] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:43.285935] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=127] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:43.319030] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=91] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:43.319065] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.319820] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.319859] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.337251] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE901-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326303336752000, start_id=1736326303336752000, end_id=1736326303336752001) [2025-01-08 00:51:43.339555] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=47] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:43.339736] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=172] temporary file module use no blocks [2025-01-08 00:51:43.339778] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=39] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:43.339795] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:43.339809] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193807253204, flush_io_finished_ret_:0, flush_io_finished_round_:1966, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:43.343685] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BD-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:43.343729] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BD-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326302846602004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326302846602004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:43.347515] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1A79C2BD-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1736326303347082000, start_id=1736326303347082000, end_id=1736326303347082001) [2025-01-08 00:51:43.347678] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F597-0-0] [lt=93] get number(ret=0, service_type_=0, range=1, base_id=1736326303347082000, start_id=1736326303347082000, end_id=1736326303347082001) [2025-01-08 00:51:43.362818] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=128] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2321, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:43.362850] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=201, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:43.362881] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=28, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:43.367236] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A6C-0-0] [lt=31] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:43.368625] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=95] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1431248, replace_num=15728) [2025-01-08 00:51:43.368654] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:43.373839] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:43.383697] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D4B-0-0] [lt=89] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:43.384343] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:43.386382] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:43.387857] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:43.388239] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:43.389580] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:43.395238] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=103] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9400, avg_group_log_size=626) [2025-01-08 00:51:43.395268] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9400) [2025-01-08 00:51:43.399815] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58594][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:43.399871] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42] worker thread created(tenant_->id()=1001, group_id_=1, token=3) [2025-01-08 00:51:43.399959] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58594][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E72-0-0] [lt=112] init point thread id with(&point=0x5572d50e61c0, idx_=1250, point=[thread id=58594, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58594) [2025-01-08 00:51:43.402393] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=77] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=139, avg_log_gen_to_submit_time=220, avg_log_submit_to_flush_time=3003, avg_log_submit_to_slide_time=3664) [2025-01-08 00:51:43.402512] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=109] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9400, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:43.405213] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:43.409303] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:43.411985] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1631, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:43.412042] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=183, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:43.412077] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=66] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=34, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:43.415927] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCD-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:43.415956] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCD-0-0] [lt=26] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:43.419316] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.419423] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:43.419472] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=43] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70914440, tenant_id=1001) [2025-01-08 00:51:43.419568] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=86] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:43.422420] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:43.422546] WDIAG read_multi_row (ob_table_access_helper.h:238) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:43.424139] INFO delete_row (ob_table_access_helper.h:339) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1001, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326303422564) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326303422564) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:43.450979] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=47] PNIO [ratelimit] time: 1736326303450976, bytes: 880950744, bw: 0.241638 MB/s, add_ts: 1007177, add_bytes: 255194 [2025-01-08 00:51:43.450999] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47CD-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:43.451061] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47CD-0-0] [lt=53] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:43.459248] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.459362] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.461353] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=74] gc stale ls task succ [2025-01-08 00:51:43.472144] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=48][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:43.472298] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=26][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:43.472346] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=12][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:43.473123] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:43.473763] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9503-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:43.473786] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9503-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4287, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:43.473797] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9503-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=138, avg_log_submit_to_flush_time=3196, avg_log_submit_to_slide_time=4149) [2025-01-08 00:51:43.473958] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:43.477949] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88D0-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:43.477989] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88D0-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4670, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:43.484778] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6B3-0-0] [lt=105][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193808398611, cache_obj->added_lc()=false, cache_obj->get_object_id()=13669, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:43.493973] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=43] ====== tenant freeze timer task ====== [2025-01-08 00:51:43.518782] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3658) [2025-01-08 00:51:43.518813] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:43.519686] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.519731] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.520324] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3658) [2025-01-08 00:51:43.520532] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=76] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3658) [2025-01-08 00:51:43.521797] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=74] clear dtl interm result cost(us)(clear_cost=8308, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:43.521939] INFO [LIB] stat (utility.h:1210) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA696F-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=31, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:43.522047] INFO [LIB] stat (utility.h:1210) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA696F-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3543, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:43.522066] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA696F-0-0] [lt=17] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=130, avg_log_gen_to_submit_time=230, avg_log_submit_to_flush_time=1900, avg_log_submit_to_slide_time=3312) [2025-01-08 00:51:43.522359] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=110] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3658, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:43.540731] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A6F-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:43.540779] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A6F-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:43.540798] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A6F-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:43.542385] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1479, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:43.542414] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:43.542436] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=129, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:43.542552] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=56, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:43.542681] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.543826] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.553387] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1469, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:43.553438] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:43.553621] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=399, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:43.553711] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:43.555010] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D9E0-0-0] [lt=38] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:43.555052] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:43.555077] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D9E0-0-0] [lt=66] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3579, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:43.555090] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58572][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D9E0-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=2009, avg_log_submit_to_slide_time=3564) [2025-01-08 00:51:43.555091] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18271, miss_cnt:0, cache_read_size:2235152, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:43.569160] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:43.574933] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:43.575121] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=241] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2649, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:43.575150] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=196, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:43.575484] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58554][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=42] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:43.575579] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:43.609760] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:43.611996] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=66, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:43.619796] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.685268] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:194) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35] get wrs ts(ls_id={id:1}, delta=336390, log_service_ts=1736326303348415, min_tx_service_ts=4611686018427387, timestamp={val:1736326303348415002, v:0}) [2025-01-08 00:51:43.685311] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=18, Other_Error_Cnt=0, Submit_Succ_Cnt=182, last_scn="{val:1736326302349610005, v:0}", last_lsn={lsn:18716034}, last_gts={val:1736326303348415000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-01-08 00:51:43.713465] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=46] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:43.717387] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5289215466, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:43.717572] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=51] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:43.717614] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=40] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5289215466, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:51:43.719941] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.731460] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=123] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:43.731494] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:43.731550] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848EA-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:43.731566] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848EA-0-0] [lt=13] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:43.761996] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=41] BlockGCTimerTask success(ret=0, cost_time_us=24, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:43.769620] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:43.771740] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:43.771836] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=62] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44940, [delay/us]:prepare:8, schedule:52, submit:48, rt:1456, total:1566) [2025-01-08 00:51:43.771853] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:43.771956] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=88] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44943, [delay/us]: prepare:8, schedule:52, submit:48, device:1456, total:1566) [2025-01-08 00:51:43.771971] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:43.772053] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:14, bw:65369, [delay/us]:prepare:7, schedule:890, submit:139, rt:1911, total:2948) [2025-01-08 00:51:43.772068] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:43.772107] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:14, avg_bw:65368, [delay/us]: prepare:7, schedule:890, submit:139, device:1911, total:2948) [2025-01-08 00:51:43.772119] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:43.772141] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:7, bw:32684, [delay/us]:prepare:6, schedule:264, submit:0, rt:1102, total:1374) [2025-01-08 00:51:43.772227] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:163422, [delay/us]:prepare:17, schedule:191, submit:99, rt:1494, total:1802) [2025-01-08 00:51:43.772242] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12256, [delay/us]:prepare:1, schedule:297, submit:197, rt:477, total:974) [2025-01-08 00:51:43.772251] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:43.772284] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4681, avg_iops:7, avg_bw:32682, [delay/us]: prepare:6, schedule:264, submit:0, device:1102, total:1374) [2025-01-08 00:51:43.772300] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12255, [delay/us]: prepare:1, schedule:297, submit:197, device:477, total:974) [2025-01-08 00:51:43.772307] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:163412, [delay/us]: prepare:17, schedule:191, submit:99, device:1494, total:1802) [2025-01-08 00:51:43.772317] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:43.772338] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4085, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:642, total:653) [2025-01-08 00:51:43.772363] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:43.772390] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:642, total:653) [2025-01-08 00:51:43.772408] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:43.772423] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:43.772431] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:43.772439] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:43.772448] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=271kB/s, net_out=218kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:43.777783] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1736326303336752004, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3, epoch_tstamp:1736326303777724}, version_delta=441030) [2025-01-08 00:51:43.801266] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=67] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:43.801437] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43485692}, readable_log_tail:{lsn:43485692}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23619076, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:43.810415] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:43.813809] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:43.813853] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:43.816278] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=78] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:43.817335] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC404-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:43.818272] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1003}, {id:1002}]) [2025-01-08 00:51:43.819222] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:202) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=6] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2025-01-08 00:51:43.819263] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=37] finish to resize log external storage handler(current_ts=1736326303819260, tenant_max_cpu=4, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:43.819279] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=14] snapshot_traffic_info once [2025-01-08 00:51:43.819488] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=13] gc stale ls task succ [2025-01-08 00:51:43.819537] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:43.819572] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=9] gc stale ls task succ [2025-01-08 00:51:43.819713] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] ====== check clog disk timer task ====== [2025-01-08 00:51:43.819760] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:43.820192] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.826722] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D09E-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:43.829061] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87139-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:43.829130] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:43.829145] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:43.829424] INFO pn_ratelimit (group.c:653) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:43.838001] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE904-0-0] [lt=34] submit log success(service_type_=0, last_id=1736326320087570000, limited_id=1736326320087570000) [2025-01-08 00:51:43.838026] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE904-0-0] [lt=25] submit log with lock success(service_type_=0, last_id=1736326320087570000, limited_id=1736326320087570000) [2025-01-08 00:51:43.841931] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=27] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=649809920, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=423931904, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:43.842030] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=59] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:43.842053] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=21] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2869, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:43.842144] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1116487680, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2104737792, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:43.842293] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=43] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:43.842312] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:43.842441] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] update id meta(ret=0, service_type=0, limited_id=1736326320087570000, latest_scn={val:1736326303837002001, v:0}, *this={tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326300843864002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326320087570000, latest_log_ts:{val:1736326303837002001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:43.842610] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=166] handle submit callback(service_type_=0, success=true, used_ts=4610, log_ts={val:1736326303837002001, v:0}, limited_id=1736326320087570000) [2025-01-08 00:51:43.842625] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=14] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:43.842634] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:43.850513] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:263) [50687][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D2008458E-0-0] [lt=28] ObArchiveSender is running(thread_index=4) [2025-01-08 00:51:43.852289] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17] get number(ret=0, service_type_=0, range=1, base_id=1736326303851673000, start_id=1736326303851673000, end_id=1736326303851673001) [2025-01-08 00:51:43.852347] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=43] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:43.853609] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=30] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:43.859624] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=91] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:43.859661] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-01-08 00:51:43.859672] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:43.859686] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:43.859700] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193807773397, flush_io_finished_ret_:0, flush_io_finished_round_:3790, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:43.871426] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185442-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:43.871761] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:43.871785] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:51:43.871793] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:43.871824] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:43.871839] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193807782830, flush_io_finished_ret_:0, flush_io_finished_round_:1962, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:43.884525] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=58] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:43.887045] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:43.888273] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:43.888400] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:43.890016] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:43.896688] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.896731] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.896849] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.896913] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:43.909140] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=58] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:43.910403] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=26] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:43.920427] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=85][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:43.944956] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18174, miss_cnt:0, cache_read_size:2223119, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:43.952128] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB614-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:43.952240] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB614-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326303837002001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326303837002001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:43.969812] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:43.978983] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58596][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:43.979059] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58596][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88DF-0-0] [lt=44] init point thread id with(&point=0x5572d50e6240, idx_=1252, point=[thread id=58596, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58596) [2025-01-08 00:51:43.979624] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:43.982449] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=48] PNIO [ratelimit] time: 1736326303982446, bytes: 0, bw: 0.000000 MB/s, add_ts: 1013347, add_bytes: 0 [2025-01-08 00:51:43.983613] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=27] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:43.986137] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2201638}], scn_array=[cnt:1, {val:1736326303851673001, v:0}], curr_size=122, accum_size=854, time_cost=12036) [2025-01-08 00:51:43.989920] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=19] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:44.010907] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=13] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:44.013259] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=52] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:44.016874] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690FCF-0-0] [lt=52] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.787115077972412109e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=893, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.792130180696641517e+01) [2025-01-08 00:51:44.021829] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.022504] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=159] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=17311626, this={dio_aligned_buf:{buf_write_offset:2515, buf_padding_size:1581, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:71007, truncate_used_ts:17485}, log_block_size:67108864, io_fd:{first_id:-1, second_id:152, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=17309696, buf_len=585, total_write_size=40238, total_write_size_after_dio=462848, ob_pwrite_used_ts=270977, count=103, each_pwrite_cost=2630) [2025-01-08 00:51:44.025442] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:44.032941] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8774, offset=2492004}, end_cursor:ObLogCursor{file_id=1, log_id=8775, offset=2492309}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c10050, len:2460, log_data_len:305, seq:8774, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:44.033308] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6B-0-0] [lt=96] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:44.033349] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6B-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326303851673000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326303851673000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:44.042365] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E6B-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326304041367000, start_id=1736326304041367000, end_id=1736326304041367001) [2025-01-08 00:51:44.048233] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8775, offset=2492309}, end_cursor:ObLogCursor{file_id=1, log_id=8776, offset=2492606}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c14050, len:2155, log_data_len:297, seq:8775, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:44.048483] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6B-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:44.048573] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6B-0-0] [lt=70] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326303851673001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326303851673001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:44.052429] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58597][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:44.052479] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58597][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B12-0-0] [lt=23] init point thread id with(&point=0x5572d50e6280, idx_=1253, point=[thread id=58597, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58597) [2025-01-08 00:51:44.053269] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1001, group_id_=1, token=4) [2025-01-08 00:51:44.056337] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=100, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:44.056482] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=142] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=8, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:44.058301] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=37] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8776, offset=2492606}, end_cursor:ObLogCursor{file_id=1, log_id=8777, offset=2492903}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c18050, len:1858, log_data_len:297, seq:8776, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:44.059353] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8777, offset=2492903}, end_cursor:ObLogCursor{file_id=1, log_id=8778, offset=2493200}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c1c050, len:1561, log_data_len:297, seq:8777, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:44.073356] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=110, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:44.073462] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:44.073525] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=68, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:44.073561] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:44.074402] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=35] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:284K/68:373072 disk:284K/68:373072 t1m2:40K/9:23279 t1m2g0:40K/9:23279 t1002m2:180K/45:363659 t1002m2g0:180K/45:363659 t1001m2:64K/14:628194 t1001m2g0:64K/14:628194 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:44.075589] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=131] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:44.085646] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:44.094266] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:19728, max_lsn:18717805, max_scn:{val:1736326303851673001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:18717805}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:18717683}, last_submit_end_lsn:{lsn:18717805}, last_submit_log_id:19728, last_submit_log_pid:1, max_flushed_lsn:{lsn:18717683}, max_flushed_end_lsn:{lsn:18717805}, max_flushed_log_pid:1, committed_end_lsn:{lsn:18717805}, last_slide_log_id:19728, last_slide_scn:{val:1736326303851673001, v:0}, last_slide_lsn:{lsn:18717683}, last_slide_end_lsn:{lsn:18717805}, last_slide_log_pid:1, last_slide_log_accum_checksum:2962187053, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8fc454530}, [StateMgr]={this:0x2ad8fc458830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:19729, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324399554770, palf_role_change_cb:0x2ad8fc45f2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8fc455c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324399758968, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/meta", dir_fd:175, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8fc45aa38, flashback_version:0}, log_storage:{log_tail:{lsn:18717805}, readable_log_tail:{lsn:18717805}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48386963, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324399554279, this:0x2ad8fc458f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324399555093, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8fc4589b0}) [2025-01-08 00:51:44.094687] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=412] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326304075667, last_flushed_end_lsn:{lsn:18717805}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326304077821, last_flushed_end_lsn:{lsn:18717805}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326304076900, last_flushed_end_lsn:{lsn:18717805}}]) [2025-01-08 00:51:44.094712] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=19729, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326304075667, last_flushed_end_lsn:{lsn:18717805}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326304077821, last_flushed_end_lsn:{lsn:18717805}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326304076900, last_flushed_end_lsn:{lsn:18717805}}]) [2025-01-08 00:51:44.094757] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1002, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18368, max_lsn:2235762, max_scn:{val:1736326303849496002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2235762}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2235640}, last_submit_end_lsn:{lsn:2235762}, last_submit_log_id:18368, last_submit_log_pid:1, max_flushed_lsn:{lsn:2235640}, max_flushed_end_lsn:{lsn:2235762}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2235762}, last_slide_log_id:18368, last_slide_scn:{val:1736326303849496002, v:0}, last_slide_lsn:{lsn:2235640}, last_slide_end_lsn:{lsn:2235762}, last_slide_log_pid:1, last_slide_log_accum_checksum:3203710836, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad92435e530}, [StateMgr]={this:0x2ad924362830, self:"192.168.159.137:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.138:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18369, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad9243692f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326301075548, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326301075548, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad92435fc70}, [ModeMgr]={palf_id:1002, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436803442001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/meta", dir_fd:184, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924364a38, flashback_version:0}, log_storage:{log_tail:{lsn:2235762}, readable_log_tail:{lsn:2235762}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64869006, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:-1, this:0x2ad924362f70}, [Reconfirm]={palf_id:1002, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9243629b0}) [2025-01-08 00:51:44.095029] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=266][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:44.095043] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:44.095053] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:44.095062] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] current log_task status(palf_id=1002, self="192.168.159.137:2882", log_id=18369, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:51:44.095347] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BB5-0-0] [lt=70] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:44.120649] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=3417, avg_group_log_size=379) [2025-01-08 00:51:44.124242] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.145977] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D9EB-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:44.146035] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D9EB-0-0] [lt=57] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3744, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:44.146050] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D9EB-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=20, avg_log_submit_to_flush_time=1735, avg_log_submit_to_slide_time=3724) [2025-01-08 00:51:44.160477] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD91-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326304274070, keepalive_ts:-1}) [2025-01-08 00:51:44.160520] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD91-0-0] [lt=41] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326304274070, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:44.163941] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB6-0-0] [lt=325][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:44.166634] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB6-0-0] [lt=47][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:44.170001] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:44.176287] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1104) [2025-01-08 00:51:44.211477] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=80] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:44.216432] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54] no need to update min/max version(new_version={val:1736326303809270000, v:0}, min_version={val:1736326303809270000, v:0}, max_version={val:1736326303809270000, v:0}) [2025-01-08 00:51:44.224799] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.246392] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:44.252919] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47DD-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:44.273628] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1226, avg_group_log_size=122) [2025-01-08 00:51:44.273670] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1226) [2025-01-08 00:51:44.273749] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:44.273766] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:44.275351] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3086, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:44.275403] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:44.275523] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=214, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:44.275594] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:44.285701] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=76, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:44.286150] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:44.286915] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9272) [2025-01-08 00:51:44.286991] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9272) [2025-01-08 00:51:44.288416] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B16-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:44.288439] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B16-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=3979, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:44.288842] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=38] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326303837002002, v:0}, new_max_version={val:1736326303837002002, v:0}, last_min_version={val:1736326303347082004, v:0}, last_max_version={val:1736326303347082004, v:0}, total_time_us=5262, delta_us=451838) [2025-01-08 00:51:44.288891] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326303837002002, v:0}, min_version={val:1736326303837002002, v:0}, max_version={val:1736326303837002002, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:44.316513] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] no need to update min/max version(new_version={val:1736326303809270000, v:0}, min_version={val:1736326303809270000, v:0}, max_version={val:1736326303809270000, v:0}) [2025-01-08 00:51:44.319931] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.319969] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.324889] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.332428] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=130] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:44.338285] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1949CA43-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1736326304337073000, start_id=1736326304337073000, end_id=1736326304337073001) [2025-01-08 00:51:44.340090] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:44.340155] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=56] temporary file module use no blocks [2025-01-08 00:51:44.340169] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:44.340232] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=59] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:44.340247] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193808253681, flush_io_finished_ret_:0, flush_io_finished_round_:1967, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:44.344432] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=53] blacklist_loop exec finished(cost_time=33, is_enabled=true, send_cnt=0) [2025-01-08 00:51:44.353760] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=511, get_gts_with_stc_cnt=3, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=3, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:44.371585] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:44.378922] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=47] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=2335, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:44.379315] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=381] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=149, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:44.379384] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=78] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=35, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:44.384764] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=57] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:44.385967] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23] current monitor number(seq_=-1) [2025-01-08 00:51:44.386377] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BE-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:44.386429] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BE-0-0] [lt=39] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326303849496005, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326303849496005, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:44.387321] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:44.389099] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:44.389300] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=65] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:44.390902] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:44.391450] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:44.403264] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:44.403302] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=33] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:44.403315] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:44.403326] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:44.406263] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:44.407133] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=864] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:44.412291] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:44.412868] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=45] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:44.413564] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58545][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B19-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=98, avg_log_submit_to_flush_time=2320, avg_log_submit_to_slide_time=5339) [2025-01-08 00:51:44.413956] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9854, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:44.418676] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCE-0-0] [lt=11] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:44.418716] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCE-0-0] [lt=37] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:44.421131] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2152, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:44.421170] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=181, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:44.421270] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:44.425192] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.455458] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47E1-0-0] [lt=48] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:44.455534] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47E1-0-0] [lt=62] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:44.456706] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=42] PNIO [ratelimit] time: 1736326304456702, bytes: 881228031, bw: 0.262936 MB/s, add_ts: 1005726, add_bytes: 277287 [2025-01-08 00:51:44.459103] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.459367] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.461490] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=69] gc stale ls task succ [2025-01-08 00:51:44.476245] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:44.477075] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1226) [2025-01-08 00:51:44.478758] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88ED-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:44.478796] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88ED-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5738, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:44.478809] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88ED-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=126, avg_log_submit_to_flush_time=3225, avg_log_submit_to_slide_time=5611) [2025-01-08 00:51:44.478938] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1226, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:44.479676] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88EE-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:44.479711] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88EE-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6703, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:44.483474] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=56] LogLoopThread round_cost_time(us)(round_cost_time=14) [2025-01-08 00:51:44.484125] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326303849496005, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=634606, in_cluster_service=true, cluster_version={val:1736326303837002002, v:0}, min_cluster_version={val:1736326303837002002, v:0}, max_cluster_version={val:1736326303837002002, v:0}, get_cluster_version_err=0, cluster_version_delta=647100, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18578, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=532401629, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:44.494909] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] LogLoopThread round_cost_time(us)(round_cost_time=25) [2025-01-08 00:51:44.505176] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6BC-0-0] [lt=29][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193809419004, cache_obj->added_lc()=false, cache_obj->get_object_id()=13670, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:44.519266] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.521534] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=84] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2953) [2025-01-08 00:51:44.521570] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:44.521986] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=30] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:44.523023] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=76] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2953) [2025-01-08 00:51:44.523144] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2953) [2025-01-08 00:51:44.525356] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=232][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.525522] INFO [LIB] stat (utility.h:1210) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6978-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=37, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:44.525598] INFO [LIB] stat (utility.h:1210) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6978-0-0] [lt=74] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=4281, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:44.525611] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58576][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6978-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=38, avg_log_gen_to_submit_time=157, avg_log_submit_to_flush_time=2398, avg_log_submit_to_slide_time=4124) [2025-01-08 00:51:44.526682] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=140] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2953, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:44.541668] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.543460] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.547097] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A83-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:44.547153] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A83-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:44.547258] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A83-0-0] [lt=101] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:44.548400] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1655, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:44.548507] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=104] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:44.548534] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=117, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:44.548666] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=68, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:44.556033] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1695, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:44.556069] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:44.556127] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=203, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:44.556377] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=88] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=58, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:44.557832] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88EF-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:44.557872] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88EF-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23829, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:44.557886] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC88EF-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=29, avg_log_submit_to_flush_time=2080, avg_log_submit_to_slide_time=23799) [2025-01-08 00:51:44.558089] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=116] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:44.558122] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18281, miss_cnt:0, cache_read_size:2236372, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:44.571772] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:44.575644] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=313] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2554, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:44.575677] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=359, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:44.575700] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:44.603469] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:112) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AE16-0-0] [lt=96] adaptive update(Tenant ID=1001, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=390, Holding Memory(MB)=102, Tenant Remain Memory(MB)=480, Usable Remain Memory(MB)=288, Last Update Limit Timestamp=1736326304600320, Is Updated=true) [2025-01-08 00:51:44.607381] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=101] get_max_applied_scn(scn={val:1736326304486211000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43495116}, palf_committed_end_scn_:{val:1736326304600320000, v:0}, last_check_scn_:{val:1736326304600320000, v:0}, max_applied_cb_scn_:{val:1736326304486211000, v:0}}) [2025-01-08 00:51:44.613346] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:44.619658] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=98, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:44.621281] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:44.621331] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=49] try gc retain ctx [2025-01-08 00:51:44.621340] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:44.621446] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=84] get_max_applied_scn(scn={val:1736326304415504000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84413957}, palf_committed_end_scn_:{val:1736326304415504001, v:0}, last_check_scn_:{val:1736326304616286000, v:0}, max_applied_cb_scn_:{val:1736326304415504000, v:0}}) [2025-01-08 00:51:44.621474] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84414543}, readable_log_tail:{lsn:84414543}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49794993, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:44.621554] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=77] finish update min start scn(max_decided_scn={val:1736326304415504000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326299381595000, v:0}, keep_alive_scn_:{val:1736326299590771001, v:0}, update_ts_:1736326304620276}) [2025-01-08 00:51:44.622370] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319254996000, latest_log_ts:{val:1736326302607470001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:44.625576] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=75] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:44.625644] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=66] try gc retain ctx [2025-01-08 00:51:44.625654] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-01-08 00:51:44.625682] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326304600320000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326294544299001, v:0}, keep_alive_scn_:{val:1736326299572894001, v:0}, update_ts_:1736326304625153}) [2025-01-08 00:51:44.626663] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=83] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319284252000, latest_log_ts:{val:1736326302668823001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:44.626890] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.628234] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BB7-0-0] [lt=48] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:44.673731] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BED3-0-0] [lt=91] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:44.708957] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=56] ====== tenant freeze timer task ====== [2025-01-08 00:51:44.714353] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=34] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:44.718419] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=58] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5291114461, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:44.718479] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:44.718495] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8dd347e60) [2025-01-08 00:51:44.718643] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=98] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5291114461, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:44.724183] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:44.724420] WDIAG read_multi_row (ob_table_access_helper.h:238) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=105][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:44.726378] INFO delete_row (ob_table_access_helper.h:339) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326304724438) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326304724438) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:44.727043] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.752411] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE3F-0-0] [lt=16] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:44.752459] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE3F-0-0] [lt=38] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:44.752492] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE3F-0-0] [lt=29] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:44.772092] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:44.772805] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=46] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:44.774738] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326304774534000, start_id=1736326304774534000, end_id=1736326304774534001) [2025-01-08 00:51:44.778938] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:44.779097] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44734, [delay/us]:prepare:8, schedule:95, submit:48, rt:1671, total:1824) [2025-01-08 00:51:44.779123] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:44.779293] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=150] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44727, [delay/us]: prepare:8, schedule:95, submit:48, device:1671, total:1824) [2025-01-08 00:51:44.779315] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:44.779370] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:81325, [delay/us]:prepare:13, schedule:99, submit:51, rt:1723, total:1887) [2025-01-08 00:51:44.779503] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=126] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:44.779569] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=49] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:81313, [delay/us]: prepare:13, schedule:99, submit:51, device:1723, total:1887) [2025-01-08 00:51:44.779586] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:44.779621] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16262, [delay/us]:prepare:9, schedule:16, submit:0, rt:925, total:951) [2025-01-08 00:51:44.779653] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:12288 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4193, iops:42, bw:174820, [delay/us]:prepare:8, schedule:388, submit:164, rt:1739, total:2301) [2025-01-08 00:51:44.779667] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1002, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:44.779716] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16263, [delay/us]: prepare:9, schedule:16, submit:0, device:925, total:951) [2025-01-08 00:51:44.779740] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4194, avg_iops:42, avg_bw:174828, [delay/us]: prepare:8, schedule:388, submit:164, device:1739, total:2301) [2025-01-08 00:51:44.779756] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:44.779787] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4065, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:308, total:318) [2025-01-08 00:51:44.779821] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:44.780119] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=290] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4064, [delay/us]: prepare:0, schedule:0, submit:600000000, device:308, total:318) [2025-01-08 00:51:44.780147] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:44.780172] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:44.780256] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=79] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:44.780271] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:44.780283] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SERVER] net_in=271kB/s, net_out=224kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:44.814128] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:44.814165] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:44.814982] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:44.820040] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:51:44.820093] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:44.820143] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=8] gc stale ls task succ [2025-01-08 00:51:44.821860] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC407-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:44.822776] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=38] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:44.824379] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:44.824611] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1}, {id:1003}]) [2025-01-08 00:51:44.827097] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.828846] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=57] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:44.829027] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=95] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326304373137000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:44.829137] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=106] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326304373137000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326304373137000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:44.829368] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=32] ====== checkpoint timer task ====== [2025-01-08 00:51:44.830518] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:44.830563] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=44] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:44.830679] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=24] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319284252000, latest_log_ts:{val:1736326302668823001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:44.830763] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=79] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:44.830795] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:44.830812] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=0, rec_log_ts={val:1736326302668823001, v:0}) [2025-01-08 00:51:44.830824] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:44.830833] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:44.830846] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:44.830865] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:44.830879] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:44.830899] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:44.830935] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=35] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:44.830949] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:44.832596] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8713F-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:44.832645] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:44.832656] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:44.836409] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0A1-0-0] [lt=27][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:44.842242] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=157] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:44.843457] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=30] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:44.843564] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326295661593000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:44.843593] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326295661593000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326295661593000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:44.853779] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=52] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:44.854502] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=47] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:44.873903] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185443-0-0] [lt=37][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:44.874275] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=93] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:44.874301] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-01-08 00:51:44.874311] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:44.874324] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:44.874337] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193808785778, flush_io_finished_ret_:0, flush_io_finished_round_:1963, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:44.883476] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=27] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:44.885410] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [46305][T1_TimerWK1][T1][YB42C0A89F89-00062B2D17685770-0-0] [lt=50] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=105) [2025-01-08 00:51:44.885694] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=59] get number(ret=0, service_type_=0, range=1, base_id=1736326304885588000, start_id=1736326304885588000, end_id=1736326304885588001) [2025-01-08 00:51:44.885938] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:44.890155] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=121] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:44.891312] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=44] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:44.892597] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=20] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:44.893851] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:44.893875] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=49] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:44.893971] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326295766421000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:44.893997] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326295766421000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326295766421000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:44.894055] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:44.896614] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.896652] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.898359] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=73] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.898461] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:44.901928] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=328] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326304685077000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=216842, in_cluster_service=true, cluster_version={val:1736326304600320000, v:0}, min_cluster_version={val:1736326304600320000, v:0}, max_cluster_version={val:1736326304600320000, v:0}, get_cluster_version_err=0, cluster_version_delta=301599, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=19015, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=650538702, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:44.911148] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=27] ====== checkpoint timer task ====== [2025-01-08 00:51:44.912076] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=40] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:44.912122] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=45] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:44.912336] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=24] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319254996000, latest_log_ts:{val:1736326302607470001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:44.912429] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=89] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319254996000, latest_log_ts:{val:1736326302607470001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:44.912488] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=52] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:44.912503] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1736326302607470001, v:0}) [2025-01-08 00:51:44.912511] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:44.912519] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:44.912533] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:44.912552] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:44.912565] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:44.912597] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:44.912627] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=29] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:44.912643] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:44.912818] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=38] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:44.913720] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:219) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:44.913824] INFO [COMMON] handle (memory_dump.cpp:538) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=101] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:44.915007] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] update_check_range(min_check_version=376, max_check_version=377, global_version=378) [2025-01-08 00:51:44.923307] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=19] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:44.925813] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=52] get number(ret=0, service_type_=0, range=1, base_id=1736326304924792000, start_id=1736326304924792000, end_id=1736326304924792001) [2025-01-08 00:51:44.929512] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:44.935640] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=25] [APPLY STAT CB APPEND COST TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=139, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:44.935679] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=38] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=5761, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:44.935688] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:44.935696] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=-766, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:44.935703] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=58, stat_interval=5000000, avg (count/cost)=5134, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:44.947143] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18184, miss_cnt:0, cache_read_size:2224330, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:44.950486] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF32-0-0] [lt=8] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326304964923, keepalive_ts:-1}) [2025-01-08 00:51:44.950572] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF32-0-0] [lt=84] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326304964923, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:44.953417] INFO handle (memory_dump.cpp:665) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 153 262 0 1 2 68 1 0 1 4 1 1 0 1 5 43 5 0 1 7 1 2 0 1 8 48 1 0 1 12 7 3 0 1 13 1 1 0 1 14 1 2 0 1 15 1 1 0 1 18 4 3 0 1 20 1 2 0 500 0 94 237 0 500 7 7 4 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 2 1 0 500 17 9 7 0 500 22 79 62 0 500 23 23 10 0 508 0 2 10 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 100 230 0 1001 2 49 1 0 1001 4 1 1 0 1001 5 22 5 0 1001 8 30 1 0 1001 12 3 2 0 1001 13 1 1 0 1001 14 1 1 0 1001 18 4 3 0 1001 20 2 2 0 1002 0 162 245 0 1002 2 59 1 0 1002 4 2 1 0 1002 5 49 6 0 1002 7 11 2 0 1002 8 47 1 0 1002 12 2 3 0 1002 13 1 1 0 1002 14 1 1 0 1002 18 9 3 0 1002 20 1 2 0 cost_time: 38452 [2025-01-08 00:51:44.953485] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=42] [MEMORY] tenant: 1, limit: 2,147,483,648 hold: 1,044,885,504 cache_hold: 213,909,504 cache_used: 213,909,504 cache_item_count: 102 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 449,880,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 142,606,336 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 95,862,784 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 100,663,296 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 14,680,064 limit= 429,496,720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:44.953875] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 449,880,064 used= 345,742,544 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,211,520 used= 31,196,160 count= 15 avg_used= 2,079,744 block_cnt= 15 chunk_cnt= 15 mod=CACHE_MAP_NODE [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 28,680,192 used= 28,531,264 count= 25 avg_used= 1,141,250 block_cnt= 25 chunk_cnt= 19 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,196,912 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 37 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,268,096 used= 11,228,160 count= 4 avg_used= 2,807,040 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,618,560 used= 7,541,760 count= 10 avg_used= 754,176 block_cnt= 10 chunk_cnt= 8 mod=ResourceGroup [MEMORY] hold= 7,004,160 used= 6,012,928 count= 121 avg_used= 49,693 block_cnt= 121 chunk_cnt= 23 mod=[T]char [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMemCU [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,440,640 used= 3,015,040 count= 140 avg_used= 21,536 block_cnt= 140 chunk_cnt= 29 mod=DiagnosticInfo [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,780,352 used= 2,663,280 count= 586 avg_used= 4,544 block_cnt= 570 chunk_cnt= 29 mod=WorkerMap [MEMORY] hold= 2,768,896 used= 2,720,000 count= 13 avg_used= 209,230 block_cnt= 13 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,640 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 272 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,394,368 used= 1,372,072 count= 75 avg_used= 18,294 block_cnt= 75 chunk_cnt= 11 mod=TabletMap [MEMORY] hold= 1,374,720 used= 1,308,288 count= 36 avg_used= 36,341 block_cnt= 36 chunk_cnt= 14 mod=CommSysVarFac [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 1,179,648 used= 1,175,808 count= 8 avg_used= 146,976 block_cnt= 8 chunk_cnt= 6 mod=SQLSessionInfo [MEMORY] hold= 960,192 used= 936,448 count= 102 avg_used= 9,180 block_cnt= 102 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 884,736 used= 434,592 count= 54 avg_used= 8,048 block_cnt= 54 chunk_cnt= 17 mod=[T]ObPerfEventR [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 497,280 used= 477,120 count= 105 avg_used= 4,544 block_cnt= 105 chunk_cnt= 26 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 262,144 used= 150,272 count= 16 avg_used= 9,392 block_cnt= 16 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 262,144 used= 261,120 count= 4 avg_used= 65,280 block_cnt= 4 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 203,840 used= 194,432 count= 49 avg_used= 3,968 block_cnt= 49 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 12 mod=SqlSession [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 13 mod=LinearHashMapNo [MEMORY] hold= 155,024 used= 2,177 count= 1,920 avg_used= 1 block_cnt= 19 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 154,432 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 13 mod=QueryAllocator [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,456 used= 131,072 count= 2 avg_used= 65,536 block_cnt= 2 chunk_cnt= 2 mod=MinorMergeMgr [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 105,664 used= 103,168 count= 13 avg_used= 7,936 block_cnt= 13 chunk_cnt= 9 mod=SqlSessiVarMap [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 97,536 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 7 mod=PlanVaIdx [MEMORY] hold= 91,664 used= 23,824 count= 337 avg_used= 70 block_cnt= 74 chunk_cnt= 25 mod=Coro [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 5 mod=HashNodeLCSta [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 54,720 used= 51,840 count= 15 avg_used= 3,456 block_cnt= 12 chunk_cnt= 11 mod=CACHE_INST [MEMORY] hold= 50,144 used= 38,976 count= 58 avg_used= 672 block_cnt= 57 chunk_cnt= 19 mod=[T]ObWarningBuf [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 38,480 used= 26,880 count= 60 avg_used= 448 block_cnt= 31 chunk_cnt= 11 mod=OMT_Worker [MEMORY] hold= 37,280 used= 3,960 count= 165 avg_used= 24 block_cnt= 66 chunk_cnt= 27 mod=[T]MemoryContex [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PartCtx [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,640 used= 15,872 count= 4 avg_used= 3,968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnMemNode [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,640 used= 6,320 count= 12 avg_used= 526 block_cnt= 11 chunk_cnt= 8 mod=SqlSessiQuerSql [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=SqlSessionSbloc [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 7,504 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 7 mod=PackStateMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 5 mod=SequenceIdMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 7 mod=ContextsMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 7 mod=SequenceMap [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,848 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 5 mod=DICollector [MEMORY] hold= 4,864 used= 4,288 count= 3 avg_used= 1,429 block_cnt= 2 chunk_cnt= 2 mod=NegParamIdex [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,168 used= 768 count= 2 avg_used= 384 block_cnt= 2 chunk_cnt= 2 mod=TableLockNode [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 2 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 320 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSIDDLVar [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 345,742,544 used= 338,174,218 count= 31,588 avg_used= 10,705 mod=SUMMARY [2025-01-08 00:51:44.953963] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=70] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 142,606,336 used= 141,492,224 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 block_cnt= 68 chunk_cnt= 68 mod=Memstore [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:44.953986] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:44.954013] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 95,862,784 used= 90,371,200 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83,884,032 used= 75,811,934 count= 2,908 avg_used= 26,070 block_cnt= 2,908 chunk_cnt= 43 mod=SqlPhyPlan [MEMORY] hold= 4,709,056 used= 4,468,656 count= 313 avg_used= 14,276 block_cnt= 313 chunk_cnt= 41 mod=SqlPlanCache [MEMORY] hold= 1,353,472 used= 1,234,048 count= 622 avg_used= 1,984 block_cnt= 208 chunk_cnt= 39 mod=CreateContext [MEMORY] hold= 422,656 used= 412,672 count= 52 avg_used= 7,936 block_cnt= 52 chunk_cnt= 12 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 90,371,200 used= 81,928,910 count= 3,897 avg_used= 21,023 mod=SUMMARY [2025-01-08 00:51:44.954033] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,368 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,272 used= 58,751 count= 259 avg_used= 226 block_cnt= 21 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,368 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-01-08 00:51:44.954046] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 100,663,296 used= 95,993,856 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 95,993,856 used= 95,815,296 count= 186 avg_used= 515,136 block_cnt= 186 chunk_cnt= 48 mod=CoStack [MEMORY] hold= 95,993,856 used= 95,815,296 count= 186 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:44.954065] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 213,909,504 used= 213,909,504 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:44.954081] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 14,680,064 used= 12,713,920 limit= 429,496,720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,036,352 used= 7,149,419 count= 109 avg_used= 65,591 block_cnt= 109 chunk_cnt= 5 mod=L_TabletPool [MEMORY] hold= 4,587,520 used= 4,569,600 count= 70 avg_used= 65,280 block_cnt= 70 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 12,713,920 used= 11,803,787 count= 181 avg_used= 65,214 mod=SUMMARY [2025-01-08 00:51:44.954120] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,584,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 block_cnt= 195 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:44.954136] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 16,256 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ExtInfoLog [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:44.954172] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=24] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 33,216 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 mod=SUMMARY [2025-01-08 00:51:44.954278] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=70] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 14,033,792 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 9,473,984 used= 9,445,184 count= 3 avg_used= 3,148,394 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 4,551,680 used= 4,444,160 count= 560 avg_used= 7,936 block_cnt= 560 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 14,033,792 used= 13,897,280 count= 564 avg_used= 24,640 mod=SUMMARY [2025-01-08 00:51:44.954312] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 73,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 73,152 used= 71,424 count= 9 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:44.954360] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 631,246,848 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 232,894,464 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 24,260,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 170,676,224 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98,557,952 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:44.954793] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 232,894,464 used= 185,994,240 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,397,376 used= 7,340,256 count= 7 avg_used= 1,048,608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7,259,504 used= 6,324,350 count= 1,032 avg_used= 6,128 block_cnt= 540 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,783,552 used= 5,752,960 count= 6 avg_used= 958,826 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 16 mod=Iteratorid()=1002, group_id_=1, token=0) [2025-01-08 00:51:44.979443] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58560][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=41] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:44.984278] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1736326304984274, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001828, add_bytes: 0 [2025-01-08 00:51:45.009584] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB615-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:45.009643] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB615-0-0] [lt=39] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326304843700000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326304843700000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:45.015463] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:45.017517] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=60] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1195366, clean_num=31457) [2025-01-08 00:51:45.019610] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690FDB-0-0] [lt=38] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.787115077972412109e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=893, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.792130180696641517e+01) [2025-01-08 00:51:45.029691] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=104][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.035722] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:45.046799] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=48] succ to submit log task to replay service(to_submit_lsn={lsn:2224386}, to_submit_scn={val:1736326304843700001, v:0}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326305046667, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2224452}, next_to_submit_scn_:{val:1736326304843700002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2224330}, end_lsn:{lsn:2224452}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2224452}, readable_log_tail:{lsn:2224452}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64880316, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326304843700001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326304843700001, v:0}, data_checksum:2124447694, flag:29775}}, init_mode_version:0, accumulate_checksum:3757417014, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326304843700001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18185, miss_cnt:0, cache_read_size:2224452, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:45.050932] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2CB-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326305065443, keepalive_ts:-1}) [2025-01-08 00:51:45.050968] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2CB-0-0] [lt=35] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326305065443, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:45.051466] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2CA-0-0] [lt=14] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326305065368, keepalive_ts:-1}) [2025-01-08 00:51:45.051573] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2CA-0-0] [lt=105] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326305065368, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:45.056527] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:45.061988] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58545][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=23] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:51:45.063389] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=26] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8778, offset=2493200}, end_cursor:ObLogCursor{file_id=1, log_id=8779, offset=2493505}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c20050, len:1264, log_data_len:305, seq:8778, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:45.063901] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6C-0-0] [lt=115] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:45.063952] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6C-0-0] [lt=31] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326304853987000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326304853987000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:45.074098] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8779, offset=2493505}, end_cursor:ObLogCursor{file_id=1, log_id=8780, offset=2493802}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c28050, len:959, log_data_len:297, seq:8779, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:45.078937] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6C-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:45.078986] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6C-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326304853987001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326304853987001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:45.079338] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=55, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:45.079375] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:45.079463] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=47, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:45.079568] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:45.081527] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=69] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/70:179143 disk:308K/70:179143 t1m2:48K/11:53325 t1m2g0:48K/11:53325 t1002m2:180K/42:141648 t1002m2g0:180K/42:141648 t1001m2:80K/17:353190 t1001m2g0:80K/17:353190 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:45.085350] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:45.087853] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58599][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:45.087911] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58599][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E8D-0-0] [lt=28] init point thread id with(&point=0x5572d50e6300, idx_=1255, point=[thread id=58599, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58599) [2025-01-08 00:51:45.088016] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread created(tenant_->id()=1001, group_id_=1, token=4) [2025-01-08 00:51:45.088331] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58600][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:45.088405] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58600][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8900-0-0] [lt=47] init point thread id with(&point=0x5572d50e6340, idx_=1256, point=[thread id=58600, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58600) [2025-01-08 00:51:45.088777] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:45.090036] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=99] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:45.092421] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=175] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8780, offset=2493802}, end_cursor:ObLogCursor{file_id=1, log_id=8781, offset=2494099}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c24050, len:662, log_data_len:297, seq:8780, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:45.096080] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=75] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326301075548, last_first_register_time_us=1736324449183670, register_time_us=1736326301075548, curr_time_us=1736326305096077) [2025-01-08 00:51:45.096144] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=33] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326305096115, last_submit_register_req_time_us=1736326305096115, register_parent_reason=1) [2025-01-08 00:51:45.096160] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:45.096317] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8781, offset=2494099}, end_cursor:ObLogCursor{file_id=1, log_id=8782, offset=2494396}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c2c050, len:4461, log_data_len:297, seq:8781, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:45.123599] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=114] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=3619, avg_group_log_size=329) [2025-01-08 00:51:45.129847] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.146518] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9535-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:45.146569] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9535-0-0] [lt=50] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12948, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:45.146583] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9535-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=23, avg_log_submit_to_flush_time=1884, avg_log_submit_to_slide_time=12925) [2025-01-08 00:51:45.146703] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=72, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:45.146729] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:45.156361] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF66-0-0] [lt=119] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326305268287, keepalive_ts:-1}) [2025-01-08 00:51:45.156423] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF66-0-0] [lt=59] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326305268287, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:45.156618] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47EF-0-0] [lt=66][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18378, log_task={header:{begin_lsn:{lsn:2236860}, end_lsn:{lsn:2236982}, log_id:18378, min_scn:{val:1736326304852832002, v:0}, max_scn:{val:1736326304852832002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2236738}, prev_proposal_id:1, committed_end_lsn:{lsn:2236860}, data_checksum:1715074492, accum_checksum:3631259340, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326305055781, freeze_ts:1736326305055781, submit_ts:1736326305055794, flushed_ts:1736326305059149, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:3355}, fs_cb_begin_ts=1736326305156607, log_life_time=100826) [2025-01-08 00:51:45.163666] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FB-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326305178025, keepalive_ts:-1}) [2025-01-08 00:51:45.163722] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FB-0-0] [lt=55] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326305178025, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:45.176966] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB7-0-0] [lt=221][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:45.177375] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:45.178844] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB7-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:45.180144] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=131] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:45.182086] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1643) [2025-01-08 00:51:45.193931] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-01-08 00:51:45.193975] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=39] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:45.211170] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=50] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:33092, max_lsn:43501944, max_scn:{val:1736326305204413000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:43501944}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:1558904}}, last_submit_lsn:{lsn:43501355}, last_submit_end_lsn:{lsn:43501944}, last_submit_log_id:33092, last_submit_log_pid:1, max_flushed_lsn:{lsn:43501355}, max_flushed_end_lsn:{lsn:43501944}, max_flushed_log_pid:1, committed_end_lsn:{lsn:43501944}, last_slide_log_id:33092, last_slide_scn:{val:1736326305204413000, v:0}, last_slide_lsn:{lsn:43501355}, last_slide_end_lsn:{lsn:43501944}, last_slide_log_pid:1, last_slide_log_accum_checksum:35473832, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad9083ee530}, [StateMgr]={this:0x2ad9083f2830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:33093, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324348477669, palf_role_change_cb:0x2ad9083f92f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad9083efc70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324348678844, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/meta", dir_fd:172, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad9083f4a38, flashback_version:0}, log_storage:{log_tail:{lsn:43501944}, readable_log_tail:{lsn:43501944}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23602824, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324348477523, this:0x2ad9083f2f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.138:2882";"192.168.159.139:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324348478003, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9083f29b0}) [2025-01-08 00:51:45.211547] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=370] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326305208644, last_flushed_end_lsn:{lsn:43501944}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326305208778, last_flushed_end_lsn:{lsn:43501944}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326305208178, last_flushed_end_lsn:{lsn:43501944}}]) [2025-01-08 00:51:45.211574] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=24] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=33093, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326305208644, last_flushed_end_lsn:{lsn:43501944}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326305208778, last_flushed_end_lsn:{lsn:43501944}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326305208178, last_flushed_end_lsn:{lsn:43501944}}]) [2025-01-08 00:51:45.218024] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:45.223236] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:45.230051] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.247534] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:45.256391] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47F1-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:45.260042] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A80-0-0] [lt=22] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326305260013}) [2025-01-08 00:51:45.260207] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A0-0-0] [lt=28] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326305260132, sql_renew_time:1736326305260132, is_mark_fail:false}]) [2025-01-08 00:51:45.260246] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A0-0-0] [lt=37] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326305260013}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326305260132, sql_renew_time:1736326305260132, is_mark_fail:false}]) [2025-01-08 00:51:45.278534] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30] start to print blacklist info [2025-01-08 00:51:45.279432] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=32] ls blacklist refresh finish(cost_time=20851) [2025-01-08 00:51:45.280511] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:51:45.280719] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=200] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:45.280791] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:45.280869] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:45.286336] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-01-08 00:51:45.286369] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:45.286479] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=115] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:45.286551] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585484-0-0] [lt=26] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:45.286579] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585484-0-0] [lt=22] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:45.287228] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3048, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:45.287300] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=70] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:45.287323] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=803, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:45.289263] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=280] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=83, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:45.289395] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=35, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:45.292062] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=44] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=11263) [2025-01-08 00:51:45.292107] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11263) [2025-01-08 00:51:45.294103] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B28-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=18, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:45.294126] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B28-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=6677, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:45.294471] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326304843700001, v:0}, new_max_version={val:1736326304843700001, v:0}, last_min_version={val:1736326304338519006, v:0}, last_max_version={val:1736326304338519006, v:0}, total_time_us=7102, delta_us=450767) [2025-01-08 00:51:45.294549] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=73] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326304843700001, v:0}, min_version={val:1736326304843700001, v:0}, max_version={val:1736326304843700001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:45.319987] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.320062] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.330141] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.340257] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=381, gc_interval=10000000) [2025-01-08 00:51:45.340329] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=46] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:45.340416] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=77] gc_check_member_list_ cost time(ret=0, time_us=112) [2025-01-08 00:51:45.340434] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=12] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:45.340447] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:45.340461] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:45.340472] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] execute_gc cost time(ret=0, time_us=12) [2025-01-08 00:51:45.340479] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] ObSafeDestroyHandler start process [2025-01-08 00:51:45.340486] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-01-08 00:51:45.340501] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-01-08 00:51:45.355415] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=89] ts response task statistics(alloc_count=60698, free_count=30348) [2025-01-08 00:51:45.377533] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:45.386208] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23] current monitor number(seq_=-1) [2025-01-08 00:51:45.386307] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:45.390274] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=52] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:45.390528] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D4C-0-0] [lt=19] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:45.391566] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=124] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:45.394130] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:45.394284] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:45.405178] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=116] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=10521, avg_group_log_size=618) [2025-01-08 00:51:45.405301] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=115] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10448) [2025-01-08 00:51:45.406439] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=3028, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:45.406476] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=397, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:45.406512] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:45.413006] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=32] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:45.418589] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:45.418834] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1736326305019847001, v:0}, base_version={val:1736326305019847000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326305219415000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326305418776, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326305019847001, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326305276076, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326305219415000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326305485140, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:45.420453] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCF-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:45.420482] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CCF-0-0] [lt=27] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:45.422392] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=91] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2309, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:45.422456] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=541, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:45.422499] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:45.430338] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.435869] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12] ====== tenant freeze timer task ====== [2025-01-08 00:51:45.435900] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70914440, tenant_id=1001) [2025-01-08 00:51:45.435923] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:45.437522] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BF-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:45.437567] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2BF-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326304852832005, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326304852832005, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:45.453743] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58597][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2E92-0-0] [lt=21] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=46, avg_log_gen_to_submit_time=150, avg_log_submit_to_flush_time=3444, avg_log_submit_to_slide_time=5201) [2025-01-08 00:51:45.453967] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10681, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:45.456623] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47F5-0-0] [lt=31] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:45.456716] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A47F5-0-0] [lt=85] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:45.458193] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=39] PNIO [ratelimit] time: 1736326305458183, bytes: 881466784, bw: 0.227356 MB/s, add_ts: 1001481, add_bytes: 238753 [2025-01-08 00:51:45.459130] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.459294] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.461589] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=41] gc stale ls task succ [2025-01-08 00:51:45.482325] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:45.483411] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A953F-0-0] [lt=107] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:45.483434] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A953F-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6262, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:45.483445] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58572][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A953F-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=375, avg_log_gen_to_submit_time=449, avg_log_submit_to_flush_time=4201, avg_log_submit_to_slide_time=5812) [2025-01-08 00:51:45.483526] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1643, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:45.483573] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC890B-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:45.483587] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC890B-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5230, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:45.484166] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:45.499280] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=63] ====== tenant freeze timer task ====== [2025-01-08 00:51:45.519404] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.524308] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:45.526306] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6BE-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193810440139, cache_obj->added_lc()=false, cache_obj->get_object_id()=13671, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:45.529480] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CEB-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4205) [2025-01-08 00:51:45.529556] INFO [LIB] stat (utility.h:1210) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CEB-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:45.529570] INFO [LIB] stat (utility.h:1210) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CEB-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4774, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:45.529580] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58576][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CEB-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=92, avg_log_submit_to_flush_time=2842, avg_log_submit_to_slide_time=4681) [2025-01-08 00:51:45.529660] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=56] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4083, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:45.530559] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.541399] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.544482] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.547490] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DA09-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:51:45.558495] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2208, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:45.558550] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:45.558576] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=178, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:45.558637] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=69, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:45.560349] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC890D-0-0] [lt=157] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:45.560387] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC890D-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=33744, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:45.560437] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC890D-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=2441, avg_log_submit_to_slide_time=33724) [2025-01-08 00:51:45.560509] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=112] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:45.560604] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=86] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18291, miss_cnt:0, cache_read_size:2237592, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:45.578597] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=90] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1446976, replace_num=15728) [2025-01-08 00:51:45.578703] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=103] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:45.582163] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=58] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3590, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:45.582252] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=70] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=511, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:45.582422] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:45.618985] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:45.621188] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=63, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:45.621300] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=95] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4205) [2025-01-08 00:51:45.622638] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4669) [2025-01-08 00:51:45.630797] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.644940] INFO [LIB] stat (utility.h:1210) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A99-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:45.645049] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58572][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3A99-0-0] [lt=96] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:45.646514] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1378, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:45.646556] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:45.646580] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=122, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:45.646640] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=69, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:45.662136] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=21] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:45.662243] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=45] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:45.662264] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=17] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=91) [2025-01-08 00:51:45.673125] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853AF-0-0] [lt=27] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:45.673231] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853AF-0-0] [lt=43] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:45.695368] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=43509144, this={dio_aligned_buf:{buf_write_offset:2021, buf_padding_size:2075, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:46122, truncate_used_ts:13502}, log_block_size:67108864, io_fd:{first_id:-1, second_id:160, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=43507712, buf_len=589, total_write_size=98317, total_write_size_after_dio=724992, ob_pwrite_used_ts=411076, count=153, each_pwrite_cost=2686) [2025-01-08 00:51:45.730864] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=80] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:45.730928] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.732712] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=29] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:45.732734] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=21] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:45.732781] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848EB-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:45.732798] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848EB-0-0] [lt=15] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:45.749670] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=28] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:45.749712] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=38] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:45.749727] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:45.749736] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=8] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:45.752309] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=5] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:51:45.753062] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=747] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:45.758448] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=17] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:45.758492] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=35] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:45.758508] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=14] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:45.758529] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:45.758539] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=9] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:45.758546] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=94) [2025-01-08 00:51:45.778843] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=189] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:45.783456] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:45.783509] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44854, [delay/us]:prepare:8, schedule:47, submit:942, rt:1207, total:2205) [2025-01-08 00:51:45.783524] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:45.783581] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44863, [delay/us]: prepare:8, schedule:47, submit:942, device:1207, total:2205) [2025-01-08 00:51:45.783639] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=51] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:45.783682] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:73411, [delay/us]:prepare:9, schedule:373, submit:350, rt:1750, total:2484) [2025-01-08 00:51:45.783693] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:45.783732] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:73422, [delay/us]: prepare:9, schedule:373, submit:350, device:1750, total:2484) [2025-01-08 00:51:45.783743] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:45.783765] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20395, [delay/us]:prepare:5, schedule:48, submit:1, rt:1795, total:1850) [2025-01-08 00:51:45.783787] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4403, iops:40, bw:175400, [delay/us]:prepare:13, schedule:104, submit:539, rt:1542, total:2200) [2025-01-08 00:51:45.783795] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:45.783823] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20396, [delay/us]: prepare:5, schedule:48, submit:1, device:1795, total:1850) [2025-01-08 00:51:45.783838] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4403, avg_iops:40, avg_bw:175408, [delay/us]: prepare:13, schedule:104, submit:539, device:1542, total:2200) [2025-01-08 00:51:45.783849] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:45.783868] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4079, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1145, total:1157) [2025-01-08 00:51:45.783891] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:45.783917] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4080, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1145, total:1157) [2025-01-08 00:51:45.783935] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:45.783949] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:45.784012] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:45.784024] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:45.784033] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=209kB/s, net_out=237kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:45.795635] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708516D-0-0] [lt=39] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:45.814010] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:45.814047] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:45.819458] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:45.820460] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=60] ====== check clog disk timer task ====== [2025-01-08 00:51:45.820487] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=21] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:45.820595] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:51:45.820634] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:45.820673] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=9] gc stale ls task succ [2025-01-08 00:51:45.821330] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87145-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:45.821377] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:45.821388] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:45.823340] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get number(ret=0, service_type_=0, range=1, base_id=1736326305823212000, start_id=1736326305823212000, end_id=1736326305823212001) [2025-01-08 00:51:45.824114] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=21] [APPLY STAT CB APPEND COST TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=158, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:45.824159] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=43] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=5150, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:45.824169] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:45.824177] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=-954, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:45.824243] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=64] [APPLY STAT CB TOTAL TIME](cur_stat_count=45, stat_interval=5000000, avg (count/cost)=4353, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:45.825399] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1002}, {id:1}]) [2025-01-08 00:51:45.829522] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=37] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:45.830139] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC40A-0-0] [lt=40][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:45.831039] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.831801] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0A4-0-0] [lt=25][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:45.835136] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58601][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:45.835368] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58602][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:45.835366] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:51:45.836441] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58601][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5CF0-0-0] [lt=27] init point thread id with(&point=0x5572d50e6380, idx_=1257, point=[thread id=58601, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58601) [2025-01-08 00:51:45.853889] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=31] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:45.856101] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326305856059], gts:1736326305854841000, latest_srr:[mts=1736326305856059]}) [2025-01-08 00:51:45.856208] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=105] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:45.856237] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326305856224], gts:1736326305854841001, latest_srr:[mts=1736326305856224]}) [2025-01-08 00:51:45.856526] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326305856511], gts:1736326305856430000, latest_srr:[mts=1736326305856511]}) [2025-01-08 00:51:45.859870] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:45.859924] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=46] temporary file module use no blocks [2025-01-08 00:51:45.859938] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:45.859952] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:45.859965] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193809773633, flush_io_finished_ret_:0, flush_io_finished_round_:3792, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:45.866977] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=58] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:45.876382] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185444-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:45.876597] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=46] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:45.876618] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-01-08 00:51:45.876626] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:45.876638] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:45.876651] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193809788228, flush_io_finished_ret_:0, flush_io_finished_round_:1964, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:45.886415] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:45.890490] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:45.892119] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:45.894422] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:45.894527] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:51:45.896562] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.896602] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.897256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.897424] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:45.914872] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:45.931157] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:45.948811] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18194, miss_cnt:0, cache_read_size:2225550, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:45.979010] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:45.988376] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1736326305856430000, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2204200}, palf_committed_end_scn_:{val:1736326305856430001, v:0}, last_check_scn_:{val:1736326305856430001, v:0}, max_applied_cb_scn_:{val:1736326305856430000, v:0}}) [2025-01-08 00:51:45.988413] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=34] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2204200}, readable_log_tail:{lsn:2204200}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64900568, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:45.991569] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=197, gc_interval=10000000) [2025-01-08 00:51:45.991614] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=24] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:45.991634] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=12] gc_check_member_list_ cost time(ret=0, time_us=40) [2025-01-08 00:51:45.991647] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:45.993267] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=6] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:45.993292] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=19] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:45.993303] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] execute_gc cost time(ret=0, time_us=13) [2025-01-08 00:51:45.993312] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:51:45.993362] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1736326305993359, bytes: 0, bw: 0.000000 MB/s, add_ts: 1009085, add_bytes: 0 [2025-01-08 00:51:46.018447] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:46.019876] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6743748608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:46.024111] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690FEA-0-0] [lt=28] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.787130217552185059e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=893, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.792035831589993933e+01) [2025-01-08 00:51:46.029335] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=32] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:46.032425] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=98][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.033434] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=11] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:46.033533] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=89] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:46.033550] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=14] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:46.033573] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=20] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:46.036692] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=24] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:46.039385] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=68] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:46.040820] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=44] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:46.040841] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=18] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:46.042298] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487F-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:46.042363] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487F-0-0] [lt=18] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:46.042378] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487F-0-0] [lt=12] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:46.042662] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852A4-0-0] [lt=14] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12208, add_timestamp:1736326306042645}) [2025-01-08 00:51:46.045124] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=12] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:46.049741] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=19] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:46.049805] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852A5-0-0] [lt=25] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12255, add_timestamp:1736326306049800}) [2025-01-08 00:51:46.051496] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=18] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:46.051516] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=19] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:46.052286] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A1-0-0] [lt=17] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12208, locations=[{table_id:12208, partition_id:12208, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306052262, sql_renew_time:1736326306052262, is_mark_fail:false}]) [2025-01-08 00:51:46.052324] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A1-0-0] [lt=36] success to process renew task(task={tenant_id:1, table_id:12208, add_timestamp:1736326306042645}, locations=[{table_id:12208, partition_id:12208, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306052262, sql_renew_time:1736326306052262, is_mark_fail:false}]) [2025-01-08 00:51:46.052354] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A2-0-0] [lt=16] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12255, locations=[{table_id:12255, partition_id:12255, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306052348, sql_renew_time:1736326306052348, is_mark_fail:false}]) [2025-01-08 00:51:46.052369] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A2-0-0] [lt=15] success to process renew task(task={tenant_id:1, table_id:12255, add_timestamp:1736326306049800}, locations=[{table_id:12255, partition_id:12255, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306052348, sql_renew_time:1736326306052348, is_mark_fail:false}]) [2025-01-08 00:51:46.054457] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487F-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:46.054584] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB616-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:46.054624] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB616-0-0] [lt=23] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326305842010001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326305842010001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:46.054766] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487F-0-0] [lt=254] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:46.054783] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487F-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:46.055036] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58603][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:46.055134] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:51:46.058356] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=11] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:46.060178] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=41] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:46.061894] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=87] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:46.061934] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=38] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:46.063322] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487F-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:46.063389] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487F-0-0] [lt=18] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:46.063402] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487F-0-0] [lt=11] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:46.063423] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=12] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:46.064023] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=596] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:46.064075] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=43] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:46.064087] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:46.064108] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487F-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:46.064127] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D1608487F-0-0] [lt=18] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:46.064137] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=7] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:46.064144] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:46.064160] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487F-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:46.064171] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D1608487F-0-0] [lt=11] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:46.064243] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=65] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:46.064250] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=7] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:46.064271] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487F-0-0] [lt=18] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:46.064285] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D1608487F-0-0] [lt=13] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:46.070944] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBE06-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193810984779, cache_obj->added_lc()=false, cache_obj->get_object_id()=13672, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:46.073610] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:46.073632] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=20] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:46.073642] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=9] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:46.073680] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:46.073694] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=13] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:46.073703] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:46.073716] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:46.073733] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D1608487F-0-0] [lt=16] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:46.081829] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=74, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:46.081869] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:46.081921] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:46.081940] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:46.082285] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:284K/66:141039 disk:284K/66:141039 t1m2:48K/11:52532 t1m2g0:48K/11:52532 t1002m2:176K/42:183602 t1002m2g0:176K/42:183602 t1001m2:60K/13:78419 t1001m2g0:60K/13:78419 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:46.090157] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=123] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:46.091458] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:46.091495] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:46.092896] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2204200}], scn_array=[cnt:1, {val:1736326305856430002, v:0}], curr_size=122, accum_size=976, time_cost=10812) [2025-01-08 00:51:46.093000] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:46.093392] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58572][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=55] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:46.095631] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58599][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=21] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:51:46.100737] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8782, offset=2494396}, end_cursor:ObLogCursor{file_id=1, log_id=8783, offset=2494701}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c34050, len:4164, log_data_len:305, seq:8782, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:46.101375] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB616-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1736326306100922000, start_id=1736326306100922000, end_id=1736326306100922001) [2025-01-08 00:51:46.103436] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6D-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:46.103474] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6D-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326305856430002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326305856430002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:46.107119] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326302078637, last_first_register_time_us=1736324454612308, register_time_us=1736326302078637, curr_time_us=1736326306107116) [2025-01-08 00:51:46.107251] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326306107147, last_submit_register_req_time_us=1736326306107147, register_parent_reason=1) [2025-01-08 00:51:46.107269] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:46.109977] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=26] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:46.114322] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8783, offset=2494701}, end_cursor:ObLogCursor{file_id=1, log_id=8784, offset=2494998}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c30050, len:3859, log_data_len:297, seq:8783, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:46.114926] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6D-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:46.115015] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6D-0-0] [lt=69] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326305856430002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326305856430002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:46.118319] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=54] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:46.121151] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=46] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5292466602, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:46.121486] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=74] recycle one thread(this=0x2ad8885abcf0, total=2, remain=1) [2025-01-08 00:51:46.121504] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5292466602, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:46.125351] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58604][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:46.125825] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=78] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4745, avg_group_log_size=431) [2025-01-08 00:51:46.126089] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=80] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8784, offset=2494998}, end_cursor:ObLogCursor{file_id=1, log_id=8785, offset=2495295}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c3c050, len:3562, log_data_len:297, seq:8784, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:46.127801] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=7] worker thread created(tenant_->id()=1001, group_id_=1, token=4) [2025-01-08 00:51:46.127909] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=27] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8785, offset=2495295}, end_cursor:ObLogCursor{file_id=1, log_id=8786, offset=2495592}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c38050, len:3265, log_data_len:297, seq:8785, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:46.135109] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.138580] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=29] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:46.148778] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9553-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:46.148805] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9553-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11487, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:46.148816] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9553-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=33, avg_log_submit_to_flush_time=2806, avg_log_submit_to_slide_time=11453) [2025-01-08 00:51:46.148948] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=13] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=55, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:46.148972] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:46.159421] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58596][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF67-0-0] [lt=23] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326306271445, keepalive_ts:-1}) [2025-01-08 00:51:46.159455] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58596][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF67-0-0] [lt=32] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326306271445, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:46.183459] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCA-0-0] [lt=9] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326306294664, keepalive_ts:-1}) [2025-01-08 00:51:46.183499] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCA-0-0] [lt=38] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326306294664, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:46.185660] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:46.193552] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:195) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB8-0-0] [lt=33][errcode=-4018] table_operator_ get_source failed(ret=-4018) [2025-01-08 00:51:46.194973] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB8-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:46.197642] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1911) [2025-01-08 00:51:46.202532] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6C0-0-0] [lt=23][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193811116366, cache_obj->added_lc()=false, cache_obj->get_object_id()=13673, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:46.209646] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=23] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=1, inc_cnt=1, queue_size=1) [2025-01-08 00:51:46.212768] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58604][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B36-0-0] [lt=2786] init point thread id with(&point=0x5572d50e6440, idx_=1260, point=[thread id=58604, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58604) [2025-01-08 00:51:46.220352] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733242368, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:46.220466] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2204322}], scn_array=[cnt:1, {val:1736326305856430003, v:0}], curr_size=122, accum_size=1098, time_cost=15370) [2025-01-08 00:51:46.227449] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1C3BBB3C-0-0] [lt=55] get number(ret=0, service_type_=0, range=1, base_id=1736326306226306000, start_id=1736326306226306000, end_id=1736326306226306001) [2025-01-08 00:51:46.229444] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB8-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:46.230097] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6C5-0-0] [lt=21] get number(ret=0, service_type_=0, range=1, base_id=1736326306229516000, start_id=1736326306229516000, end_id=1736326306229516001) [2025-01-08 00:51:46.235334] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.239051] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:51:46.239054] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58608][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:46.244315] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18721481}], scn_array=[cnt:1, {val:1736326306226306000, v:0}], curr_size=545, accum_size=2090, time_cost=16621) [2025-01-08 00:51:46.249568] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=91][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84421369}], scn_array=[cnt:1, {val:1736326306224286000, v:0}], curr_size=586, accum_size=3329, time_cost=23623) [2025-01-08 00:51:46.251509] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=60][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43511144}], scn_array=[cnt:1, {val:1736326306208919000, v:0}], curr_size=589, accum_size=9200, time_cost=41954) [2025-01-08 00:51:46.253836] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:46.260133] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4805-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:46.291685] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get number(ret=0, service_type_=0, range=1, base_id=1736326306288209000, start_id=1736326306288209000, end_id=1736326306288209001) [2025-01-08 00:51:46.291890] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=77, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:46.293257] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:46.295260] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=32] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9789) [2025-01-08 00:51:46.295318] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=60] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9789) [2025-01-08 00:51:46.296505] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B37-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=12, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:46.296527] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B37-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=6582, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:46.297142] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326305842010002, v:0}, new_max_version={val:1736326305842010002, v:0}, last_min_version={val:1736326305353857003, v:0}, last_max_version={val:1736326305353857003, v:0}, total_time_us=8059, delta_us=455129) [2025-01-08 00:51:46.297175] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326305842010002, v:0}, min_version={val:1736326305842010002, v:0}, max_version={val:1736326305842010002, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:46.304380] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:46.304408] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:46.305943] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4278, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:46.305970] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:46.306036] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1072, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:46.306073] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=66, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:46.307165] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=93, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:46.307254] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=88] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=5783, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:46.307265] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:46.307273] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=-800, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:46.307280] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=5076, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:46.310304] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1003, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18287, max_lsn:2225916, max_scn:{val:1736326305842010004, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2225916}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2225794}, last_submit_end_lsn:{lsn:2225916}, last_submit_log_id:18287, last_submit_log_pid:1, max_flushed_lsn:{lsn:2225794}, max_flushed_end_lsn:{lsn:2225916}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2225794}, last_slide_log_id:18286, last_slide_scn:{val:1736326305842010003, v:0}, last_slide_lsn:{lsn:2225672}, last_slide_end_lsn:{lsn:2225794}, last_slide_log_pid:1, last_slide_log_accum_checksum:343924652, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924fd6530}, [StateMgr]={this:0x2ad924fda830, self:"192.168.159.137:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.139:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18287, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad924fe12f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326306107147, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326306107147, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924fd7c70}, [ModeMgr]={palf_id:1003, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436825152001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/meta", dir_fd:181, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924fdca38, flashback_version:0}, log_storage:{log_tail:{lsn:2225916}, readable_log_tail:{lsn:2225916}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64878852, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, palf_epoch:4, last_purge_throttling_ts:-1, this:0x2ad924fdaf70}, [Reconfirm]={palf_id:1003, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924fda9b0}) [2025-01-08 00:51:46.310623] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=311][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:46.310640] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:46.310652] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:46.310662] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] current log_task status(palf_id=1003, self="192.168.159.137:2882", log_id=18287, log_task={header:{begin_lsn:{lsn:2225794}, end_lsn:{lsn:2225916}, log_id:18287, min_scn:{val:1736326305842010004, v:0}, max_scn:{val:1736326305842010004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2225672}, prev_proposal_id:1, committed_end_lsn:{lsn:2225794}, data_checksum:940009244, accum_checksum:2990806468, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1736326306252018, freeze_ts:1736326306252018, submit_ts:1736326306252023, flushed_ts:1736326306253293, gen_to_freeze cost time:0, gen_to_submit cost time:5, submit_to_flush cost time:1270}, ack_info_list=[]) [2025-01-08 00:51:46.310739] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A3-0-0] [lt=12] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12274, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:46.311546] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A3-0-0] [lt=18] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12274, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306311523, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306311523, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306311523, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:46.311589] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A3-0-0] [lt=42] success to process renew task(task={tenant_id:1, table_id:12274, add_timestamp:1736326306309948}, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306311523, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306311523, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326306311523, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:46.319074] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.319122] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.324213] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326306024396000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=299805, in_cluster_service=true, cluster_version={val:1736326306024396000, v:0}, min_cluster_version={val:1736326306024396000, v:0}, max_cluster_version={val:1736326306024396000, v:0}, get_cluster_version_err=0, cluster_version_delta=299805, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36767, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=560821188, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:46.335554] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:46.335653] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=95][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.341114] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=40] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:46.341160] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=39] temporary file module use no blocks [2025-01-08 00:51:46.341172] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:46.341239] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:46.341260] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193810254465, flush_io_finished_ret_:0, flush_io_finished_round_:1969, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:46.346869] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:46.356920] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=621, get_gts_with_stc_cnt=33, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=31, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:46.356966] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=34] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=529, get_gts_with_stc_cnt=77, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=75, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:46.376133] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C2BB62A-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1736326306373752000, start_id=1736326306373752000, end_id=1736326306373752001) [2025-01-08 00:51:46.376133] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D16EBBDC2-0-0] [lt=56] get number(ret=0, service_type_=0, range=1, base_id=1736326306373752000, start_id=1736326306373752000, end_id=1736326306373752001) [2025-01-08 00:51:46.385881] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:46.386404] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2025-01-08 00:51:46.386873] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:46.390678] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=55] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:46.392366] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=195] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:46.394616] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:46.394752] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=91] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:46.404241] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [50463][T1002_ArcFetche][T1002][YB42C0A89F89-00062B2D1C18458E-0-0] [lt=118] ObArchiveFetcher is running(thread_index=0) [2025-01-08 00:51:46.404526] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=2456, avg_group_log_size=223) [2025-01-08 00:51:46.404690] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=158] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2456) [2025-01-08 00:51:46.406979] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB APPEND COST TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=185, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:46.407008] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=28] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=7735, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:46.407018] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:46.407026] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=-1305, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:46.407033] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=6615, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:46.409866] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=10335, avg_group_log_size=645) [2025-01-08 00:51:46.409935] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=65] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10335) [2025-01-08 00:51:46.410874] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4382, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:46.410902] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=189, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:46.410928] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=22, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:46.415075] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=32] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:46.420882] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:46.422427] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD0-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:46.422481] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD0-0-0] [lt=51] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:46.427306] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3123, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:46.427339] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=419, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:46.427369] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=71] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=21, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:46.429093] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:46.435913] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.459210] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.459365] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.460315] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1736326306460312, bytes: 881751825, bw: 0.271259 MB/s, add_ts: 1002129, add_bytes: 285041 [2025-01-08 00:51:46.460454] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4809-0-0] [lt=26] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:46.460494] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4809-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:46.460531] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4809-0-0] [lt=16][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18391, log_task={header:{begin_lsn:{lsn:2238437}, end_lsn:{lsn:2238559}, log_id:18391, min_scn:{val:1736326306356405000, v:0}, max_scn:{val:1736326306356405000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2238315}, prev_proposal_id:1, committed_end_lsn:{lsn:2238437}, data_checksum:1715074492, accum_checksum:3169084121, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326306360473, freeze_ts:1736326306360473, submit_ts:1736326306360486, flushed_ts:1736326306364653, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:4167}, fs_cb_begin_ts=1736326306460519, log_life_time=100046) [2025-01-08 00:51:46.461791] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:46.468308] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C0-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:46.468391] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C0-0-0] [lt=62] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326306356405001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326306356405001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:46.483335] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B3A-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=81, avg_log_gen_to_submit_time=168, avg_log_submit_to_flush_time=4589, avg_log_submit_to_slide_time=6104) [2025-01-08 00:51:46.483529] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10336, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:46.485823] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58609][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:46.485897] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:51:46.506159] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2334) [2025-01-08 00:51:46.507407] INFO [LIB] stat (utility.h:1210) [58596][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A955D-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:46.507435] INFO [LIB] stat (utility.h:1210) [58596][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A955D-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6612, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:46.507446] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58596][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A955D-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=153, avg_log_gen_to_submit_time=248, avg_log_submit_to_flush_time=4350, avg_log_submit_to_slide_time=6363) [2025-01-08 00:51:46.507668] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=343] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2334, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:46.508435] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC892A-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:46.508457] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC892A-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6835, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:46.508580] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:46.519076] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=49] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.528460] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:46.529368] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=5611) [2025-01-08 00:51:46.530780] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58602][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA698F-0-0] [lt=21] init point thread id with(&point=0x5572d50e63c0, idx_=1258, point=[thread id=58602, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58602) [2025-01-08 00:51:46.530955] INFO [LIB] stat (utility.h:1210) [58602][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA698F-0-0] [lt=46] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=21, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:46.530971] INFO [LIB] stat (utility.h:1210) [58602][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA698F-0-0] [lt=15] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=7706, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:46.530980] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58602][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA698F-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=1629, avg_log_gen_to_submit_time=1712, avg_log_submit_to_flush_time=3553, avg_log_submit_to_slide_time=5993) [2025-01-08 00:51:46.531150] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5733, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:46.536082] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.543362] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.544301] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.546426] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6C6-0-0] [lt=20][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193811460261, cache_obj->added_lc()=false, cache_obj->get_object_id()=13674, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:46.549246] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A955F-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:46.561807] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1880, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:46.561849] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:46.561871] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=810, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:46.562008] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=66, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:46.563578] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC892B-0-0] [lt=45] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:46.563618] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC892B-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14371, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:46.563631] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC892B-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=12, avg_log_submit_to_flush_time=2728, avg_log_submit_to_slide_time=14358) [2025-01-08 00:51:46.563696] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:46.563722] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18301, miss_cnt:0, cache_read_size:2238803, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:46.580648] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=16] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:46.586112] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:46.606070] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3381, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:46.606122] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=120, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:46.606157] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=35, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:46.615751] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58610][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:46.616168] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:46.616172] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58611][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:46.621484] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:46.626571] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=59, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:46.626622] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5733) [2025-01-08 00:51:46.627647] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5733) [2025-01-08 00:51:46.636291] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.646418] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AAD-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:46.646456] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AAD-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:46.647525] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2290, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:46.647552] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:46.647569] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=906, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:46.647623] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=64] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=56, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:46.694275] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B3D-0-0] [lt=46] init point thread id with(&point=0x5572d50e65c0, idx_=1266, point=[thread id=58609, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58609) [2025-01-08 00:51:46.706534] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1A7-0-0] [lt=16] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:46.706574] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1A7-0-0] [lt=33] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:46.706605] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1A7-0-0] [lt=28] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:46.707412] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [50457][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=26] XA scheduler heartbeat task statistics(avg_time=2) [2025-01-08 00:51:46.720100] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=102] ====== tenant freeze timer task ====== [2025-01-08 00:51:46.736439] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.786443] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=99] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:46.796558] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:46.796641] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:52557, [delay/us]:prepare:10, schedule:208, submit:1457, rt:1371, total:3047) [2025-01-08 00:51:46.796660] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:46.796718] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52557, [delay/us]: prepare:10, schedule:208, submit:1457, device:1371, total:3047) [2025-01-08 00:51:46.796730] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:46.796768] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4915, iops:15, bw:72775, [delay/us]:prepare:7, schedule:368, submit:760, rt:1089, total:2225) [2025-01-08 00:51:46.796806] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:46.796849] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4915, avg_iops:15, avg_bw:72773, [delay/us]: prepare:7, schedule:368, submit:760, device:1089, total:2225) [2025-01-08 00:51:46.796861] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:46.796884] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20214, [delay/us]:prepare:6, schedule:20, submit:1, rt:1287, total:1315) [2025-01-08 00:51:46.796905] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:38, bw:153632, [delay/us]:prepare:11, schedule:728, submit:314, rt:1483, total:2538) [2025-01-08 00:51:46.796913] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:46.796964] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20214, [delay/us]: prepare:6, schedule:20, submit:1, device:1287, total:1315) [2025-01-08 00:51:46.796982] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:38, avg_bw:153628, [delay/us]: prepare:11, schedule:728, submit:314, device:1483, total:2538) [2025-01-08 00:51:46.796991] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:46.797010] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4042, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:288, total:358) [2025-01-08 00:51:46.797055] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:46.797088] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4042, [delay/us]: prepare:0, schedule:0, submit:600000000, device:288, total:358) [2025-01-08 00:51:46.797107] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:46.797121] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:46.797129] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:46.797147] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:46.797156] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=285kB/s, net_out=284kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:46.803743] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC40B-0-0] [lt=35] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326306803075, can_tenant_elr:true}) [2025-01-08 00:51:46.814950] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:46.815025] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=72][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:46.817444] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58610][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9567-0-0] [lt=249] init point thread id with(&point=0x5572d50e6600, idx_=1267, point=[thread id=58610, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58610) [2025-01-08 00:51:46.819624] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC40D-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:46.820870] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:46.820956] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:46.821000] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=9] gc stale ls task succ [2025-01-08 00:51:46.822343] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=101] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:46.824846] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8714B-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:46.824946] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=69][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:46.824961] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:46.827664] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=66] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1002}, {id:1003}]) [2025-01-08 00:51:46.829778] INFO pn_ratelimit (group.c:653) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=10] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:46.830693] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0A7-0-0] [lt=349][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:46.830964] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=48] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:46.836605] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.837242] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=46] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:46.837272] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=21] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=177) [2025-01-08 00:51:46.837778] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=53] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:46.837869] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=82] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:46.837886] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=16] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=193) [2025-01-08 00:51:46.842595] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=69] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=649809920, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=423931904, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:46.842746] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=64] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:46.842768] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=20] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2875, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:46.842863] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=38] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1116487680, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2104737792, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:46.842945] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=48] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:46.842960] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:46.844411] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE914-0-0] [lt=191] get number(ret=0, service_type_=0, range=1, base_id=1736326306844189000, start_id=1736326306844189000, end_id=1736326306844189001) [2025-01-08 00:51:46.854021] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=34] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:46.857713] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:46.860196] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=51] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:46.860228] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-01-08 00:51:46.860294] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=62] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:46.860311] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:46.860324] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193810773861, flush_io_finished_ret_:0, flush_io_finished_round_:3793, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:46.878590] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185445-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:46.887119] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:46.893078] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:46.893450] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=38] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-01-08 00:51:46.894982] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:46.895266] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=70] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:46.896457] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.897363] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.897401] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.897467] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:46.918150] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:46.936853] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:46.950857] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18204, miss_cnt:0, cache_read_size:2226770, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:46.965738] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE8-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:46.965796] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE8-0-0] [lt=49] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=62) [2025-01-08 00:51:46.986758] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:46.990252] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=75] get_min_unreplayed_log_info(lsn={lsn:2239291}, scn={val:1736326306859288002, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326306964840, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2239291}, next_to_submit_scn_:{val:1736326306859288002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2239169}, end_lsn:{lsn:2239291}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2239291}, readable_log_tail:{lsn:2239291}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64865477, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326306859288001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326306859288001, v:0}, data_checksum:3348837161, flag:21839}}, init_mode_version:0, accumulate_checksum:906516160, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326306859288001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18305, miss_cnt:0, cache_read_size:2239291, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:46.999524] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1736326306999520, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006161, add_bytes: 0 [2025-01-08 00:51:47.005666] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BB7-0-0] [lt=18] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:47.016637] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58612][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:47.016731] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:51:47.022847] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:47.024497] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=54] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1226823, clean_num=31457) [2025-01-08 00:51:47.027309] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690FF9-0-0] [lt=15] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.787130217552185059e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=893, Throttle Trigger(MB)=536, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.792035831589993933e+01) [2025-01-08 00:51:47.036947] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [46323][T1_RFLWorker][T1][YB42C0A89F89-00062B2D1618458E-0-0] [lt=87] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:51:47.036986] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.082150] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:47.082243] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=82] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=87) [2025-01-08 00:51:47.090354] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=117] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326306356405004, v:0}, base_version={val:1736326306344258002, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326306844189001, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326307090334, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326306356405004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326307173108, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326306356405005, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326307079634, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:47.092375] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:47.095658] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get_max_applied_scn(scn={val:1736326306985404000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18723303}, palf_committed_end_scn_:{val:1736326306985404000, v:0}, last_check_scn_:{val:1736326306985404000, v:0}, max_applied_cb_scn_:{val:1736326306985404000, v:0}}) [2025-01-08 00:51:47.095697] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=34] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18723303}, readable_log_tail:{lsn:18723303}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48381465, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:47.095794] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=76] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:47.103551] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=24] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84426198}, readable_log_tail:{lsn:84426198}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49783338, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:47.103634] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=82] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:47.103668] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=11] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:47.106313] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=62, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:47.106361] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:47.106509] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:47.106546] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:47.106624] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=100] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:300K/70:479460 disk:300K/70:479460 t1m2:48K/11:196164 t1m2g0:48K/11:196164 t1002m2:180K/43:534521 t1002m2g0:180K/43:534521 t1001m2:72K/16:526249 t1001m2g0:72K/16:526249 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:47.108076] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:47.120514] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=30] LogLoopThread round_cost_time(us)(round_cost_time=13) [2025-01-08 00:51:47.122861] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5294207440, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:47.122936] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:47.122952] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a0030) [2025-01-08 00:51:47.122961] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5294207440, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:47.130909] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=198, gc_interval=10000000) [2025-01-08 00:51:47.130962] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=30] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:47.130978] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=12] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:47.131894] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4951, avg_group_log_size=450) [2025-01-08 00:51:47.132065] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=25] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8786, offset=2495592}, end_cursor:ObLogCursor{file_id=1, log_id=8787, offset=2495897}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c40050, len:2968, log_data_len:305, seq:8786, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:47.134089] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6E-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:47.134134] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6E-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326306985404000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326306985404000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:47.134720] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:51:47.134942] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB617-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:47.134978] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB617-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326306844189002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326306844189002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:47.136116] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=41] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:51:47.136156] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=36] gc_check_member_list_ cost time(ret=0, time_us=5216) [2025-01-08 00:51:47.136316] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:47.139329] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=38] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:47.140904] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=45] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:47.142591] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=38] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:47.144544] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=46] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:47.144590] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=39] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:47.144609] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:47.144620] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:47.144631] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:47.144642] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=54) [2025-01-08 00:51:47.144650] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:51:47.144771] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.150995] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9571-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:47.151015] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9571-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30873, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:47.151025] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9571-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=20, avg_log_submit_to_flush_time=2091, avg_log_submit_to_slide_time=30853) [2025-01-08 00:51:47.151637] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=360] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:47.151660] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:47.157533] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58612][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B42-0-0] [lt=30] init point thread id with(&point=0x5572d50e6640, idx_=1268, point=[thread id=58612, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58612) [2025-01-08 00:51:47.159420] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=48] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8787, offset=2495897}, end_cursor:ObLogCursor{file_id=1, log_id=8788, offset=2496194}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c44050, len:2663, log_data_len:297, seq:8787, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:47.160100] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6E-0-0] [lt=21] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:47.160160] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6E-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326306985404000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326306985404000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:47.164830] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=18] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:47.172534] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8788, offset=2496194}, end_cursor:ObLogCursor{file_id=1, log_id=8789, offset=2496491}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c48050, len:2366, log_data_len:297, seq:8788, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:47.173904] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8789, offset=2496491}, end_cursor:ObLogCursor{file_id=1, log_id=8790, offset=2496788}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c4c050, len:2069, log_data_len:297, seq:8789, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:47.186975] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:47.194255] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=28] ====== check clog disk timer task ====== [2025-01-08 00:51:47.194297] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=33] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:47.202369] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46503][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6C8-0-0] [lt=35][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193812116200, cache_obj->added_lc()=false, cache_obj->get_object_id()=13675, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:47.207922] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:51:47.211450] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] no need to update min/max version(new_version={val:1736326306808599000, v:0}, min_version={val:1736326306808599000, v:0}, max_version={val:1736326306808599000, v:0}) [2025-01-08 00:51:47.226648] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:47.231915] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=133] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:47.234205] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58602][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=10] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-01-08 00:51:47.239278] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CB9-0-0] [lt=73][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:47.240985] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CB9-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:47.243662] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] ====== traversal_flush timer task ====== [2025-01-08 00:51:47.243768] INFO [MDS] flush (mds_table_mgr.cpp:195) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585489-0-0] [lt=31] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326306985404001, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326320087570000, latest_log_ts:{val:1736326303837002001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.243981] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585489-0-0] [lt=178] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326302850005000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.244065] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585489-0-0] [lt=56] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326302846602000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.244295] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585489-0-0] [lt=72] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326302835077000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.244360] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585489-0-0] [lt=62] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:47.244992] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.263847] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4819-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:47.268951] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:47.286486] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2025-01-08 00:51:47.286538] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=43] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:47.292909] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=63, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:47.293878] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:47.296535] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58609][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EAE-0-0] [lt=64] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9777) [2025-01-08 00:51:47.297046] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9777) [2025-01-08 00:51:47.297305] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=13, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:47.297425] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=118] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=5046, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:47.298111] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326306844189002, v:0}, new_max_version={val:1736326306844189002, v:0}, last_min_version={val:1736326306356405004, v:0}, last_max_version={val:1736326306356405004, v:0}, total_time_us=7392, delta_us=453919) [2025-01-08 00:51:47.298168] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=52] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326306844189002, v:0}, min_version={val:1736326306844189002, v:0}, max_version={val:1736326306844189002, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:47.307068] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:47.307120] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:47.310287] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1397, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:47.310406] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=116] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:47.310456] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=103, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:47.310491] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:47.314072] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326307313366000, start_id=1736326307313366000, end_id=1736326307313366001) [2025-01-08 00:51:47.319315] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.319375] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.345079] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.346019] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE919-0-0] [lt=9] get number(ret=0, service_type_=0, range=1, base_id=1736326307345818000, start_id=1736326307345818000, end_id=1736326307345818001) [2025-01-08 00:51:47.351937] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:47.358431] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1829F5AE-0-0] [lt=92] get number(ret=0, service_type_=0, range=1, base_id=1736326307357424000, start_id=1736326307357424000, end_id=1736326307357424001) [2025-01-08 00:51:47.366903] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2243753, this={dio_aligned_buf:{buf_write_offset:3363, buf_padding_size:733, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:26925, truncate_used_ts:5506}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2240512, buf_len=122, total_write_size=12294, total_write_size_after_dio=425984, ob_pwrite_used_ts=238940, count=101, each_pwrite_cost=2365) [2025-01-08 00:51:47.386618] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=37] current monitor number(seq_=-1) [2025-01-08 00:51:47.387255] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:47.387558] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:47.393275] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=125] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:47.393675] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:47.395249] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:47.395364] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=50] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:47.399033] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D4D-0-0] [lt=45] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:47.405545] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=27] ====== tenant freeze timer task ====== [2025-01-08 00:51:47.406885] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:51:47.406917] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:47.418523] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:47.424421] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD1-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:47.424468] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD1-0-0] [lt=43] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:47.427294] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:47.430825] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=32] ====== traversal_flush timer task ====== [2025-01-08 00:51:47.430834] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1249, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:47.430881] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=128, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:47.431039] INFO [MDS] flush (mds_table_mgr.cpp:195) [50558][T1001_TimerWK0][T1001][YB42C0A89F89-00062B2D1E484E4F-0-0] [lt=24] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326307313366000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319284252000, latest_log_ts:{val:1736326302668823001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.431142] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50558][T1001_TimerWK0][T1001][YB42C0A89F89-00062B2D1E484E4F-0-0] [lt=96] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:47.431281] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:47.434494] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=34] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:47.445533] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.454337] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:47.454372] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70914440, tenant_id=1001) [2025-01-08 00:51:47.454395] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:47.459229] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=58] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.459460] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.460857] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1736326307460854, bytes: 882011069, bw: 0.247100 MB/s, add_ts: 1000542, add_bytes: 259244 [2025-01-08 00:51:47.462768] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=27] gc stale ls task succ [2025-01-08 00:51:47.464913] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A481D-0-0] [lt=27] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:47.464975] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A481D-0-0] [lt=47] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:47.475553] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:47.475648] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=92] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:47.475665] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:47.475676] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:47.478432] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=132, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}]) [2025-01-08 00:51:47.478884] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=448] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:47.485703] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB02E-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326307484744000, start_id=1736326307484744000, end_id=1736326307484744001) [2025-01-08 00:51:47.493109] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9819, avg_group_log_size=654) [2025-01-08 00:51:47.493140] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9820) [2025-01-08 00:51:47.494822] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=1943, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:47.494870] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=227, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:47.496247] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=24, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:47.496872] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C1-0-0] [lt=95] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:47.496930] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C1-0-0] [lt=39] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326307359912001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326307359912001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:47.497513] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58604][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B48-0-0] [lt=102] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=85, avg_log_submit_to_flush_time=2400, avg_log_submit_to_slide_time=5394) [2025-01-08 00:51:47.497760] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9069, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:47.508854] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:51:47.511929] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A957B-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:47.511955] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A957B-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5142, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:47.512112] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8948-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:47.512125] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8948-0-0] [lt=12] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5226, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:47.512133] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8948-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=102, avg_log_submit_to_flush_time=2057, avg_log_submit_to_slide_time=5123) [2025-01-08 00:51:47.512525] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58611][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A957C-0-0] [lt=21] init point thread id with(&point=0x5572d50e6680, idx_=1269, point=[thread id=58611, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58611) [2025-01-08 00:51:47.514774] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:47.514823] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1643, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:47.518263] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.533181] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:47.534310] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3963) [2025-01-08 00:51:47.535927] INFO [LIB] stat (utility.h:1210) [58601][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D01-0-0] [lt=56] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:47.535948] INFO [LIB] stat (utility.h:1210) [58601][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D01-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4382, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:47.535959] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58601][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D01-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=96, avg_log_submit_to_flush_time=1515, avg_log_submit_to_slide_time=4285) [2025-01-08 00:51:47.536022] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=76] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3963, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:47.542359] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.544312] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.545638] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.550342] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3ABF-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:47.570115] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6CA-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193812483946, cache_obj->added_lc()=false, cache_obj->get_object_id()=13676, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:47.570167] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1837, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:47.570285] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=110] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=97, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:47.570381] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=65, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:47.587896] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=122] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:47.608497] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=111] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1905, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:47.608612] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=107] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=91, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:47.608685] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:47.612843] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:47.613740] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58538][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:47.627873] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=92] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:47.629940] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=64, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:47.629986] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3963) [2025-01-08 00:51:47.630921] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3962) [2025-01-08 00:51:47.632964] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:47.645876] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.646097] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BB9-0-0] [lt=48] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:47.649520] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AC1-0-0] [lt=94] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:47.649718] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AC1-0-0] [lt=186] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:47.651061] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1542, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:47.651123] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:47.651214] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=269, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:47.651292] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:47.664714] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4821-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:47.664757] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4821-0-0] [lt=41] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21913, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:47.664770] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4821-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=1988, avg_log_submit_to_slide_time=21898) [2025-01-08 00:51:47.664738] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:47.664800] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=55] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18311, miss_cnt:0, cache_read_size:2240023, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:47.665775] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:47.671705] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BED4-0-0] [lt=93] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:47.695958] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:47.696002] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=41] get_min_unreplayed_log_info(lsn={lsn:2227624}, scn={val:1736326307584911001, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326307653278, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2227624}, next_to_submit_scn_:{val:1736326307584911001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2227502}, end_lsn:{lsn:2227624}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2227624}, readable_log_tail:{lsn:2227624}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64877144, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326307584911000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326307584911000, v:0}, data_checksum:2124447694, flag:46887}}, init_mode_version:0, accumulate_checksum:2284970566, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326307584911000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18211, miss_cnt:0, cache_read_size:2227624, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:47.707257] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:47.707285] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] try gc retain ctx [2025-01-08 00:51:47.707293] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-01-08 00:51:47.707317] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326307358778002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326297338357003, v:0}, keep_alive_scn_:{val:1736326302349610005, v:0}, update_ts_:1736326307706633}) [2025-01-08 00:51:47.708055] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326320087570000, latest_log_ts:{val:1736326303837002001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:47.708407] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=201] finish update min start scn(max_decided_scn={val:1736326307358778002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326297338357003, v:0}, keep_alive_scn_:{val:1736326302349610004, v:0}, update_ts_:1736326307708304}) [2025-01-08 00:51:47.709089] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326302850005000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:47.709331] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] finish update min start scn(max_decided_scn={val:1736326307593236000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326300341303005, v:0}, keep_alive_scn_:{val:1736326305854841000, v:0}, update_ts_:1736326307708304}) [2025-01-08 00:51:47.710048] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326302846602000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:47.710115] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=57] finish update min start scn(max_decided_scn={val:1736326307584911000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326299327803001, v:0}, keep_alive_scn_:{val:1736326304338519003, v:0}, update_ts_:1736326307709483}) [2025-01-08 00:51:47.710978] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326302835077000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:47.733989] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848EC-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:47.734018] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848EC-0-0] [lt=26] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:47.733929] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:47.734036] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=106] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:47.746012] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.788964] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=28] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1462704, replace_num=15728) [2025-01-08 00:51:47.789066] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=101] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:47.801799] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:47.801853] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44822, [delay/us]:prepare:7, schedule:42, submit:69, rt:1097, total:1216) [2025-01-08 00:51:47.801870] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:47.801927] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44822, [delay/us]: prepare:7, schedule:42, submit:69, device:1097, total:1216) [2025-01-08 00:51:47.801940] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:47.801980] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:73345, [delay/us]:prepare:10, schedule:396, submit:441, rt:1381, total:2230) [2025-01-08 00:51:47.801990] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:47.802026] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:73348, [delay/us]: prepare:10, schedule:396, submit:441, device:1381, total:2230) [2025-01-08 00:51:47.802347] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=313] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:47.802373] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16294, [delay/us]:prepare:5, schedule:140, submit:1, rt:1125, total:1272) [2025-01-08 00:51:47.802397] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4198, iops:40, bw:167019, [delay/us]:prepare:6, schedule:113, submit:317, rt:1384, total:1821) [2025-01-08 00:51:47.802407] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:47.802440] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16294, [delay/us]: prepare:5, schedule:140, submit:1, device:1125, total:1272) [2025-01-08 00:51:47.802456] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4198, avg_iops:40, avg_bw:167021, [delay/us]: prepare:6, schedule:113, submit:317, device:1384, total:1821) [2025-01-08 00:51:47.802466] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:47.802486] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4073, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:341, total:353) [2025-01-08 00:51:47.802509] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:47.802536] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4073, [delay/us]: prepare:0, schedule:0, submit:600000000, device:341, total:353) [2025-01-08 00:51:47.802553] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:47.802568] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:47.802577] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:47.802585] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:47.802593] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=349kB/s, net_out=243kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:47.813938] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=48] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:47.817403] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC410-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:47.817724] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:47.817807] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:47.820622] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=124] ====== check clog disk timer task ====== [2025-01-08 00:51:47.820648] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=21] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:47.821795] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=55] gc stale ls task succ [2025-01-08 00:51:47.821836] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:47.821875] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=32] gc stale ls task succ [2025-01-08 00:51:47.827396] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87151-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:47.827471] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:47.827486] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:47.827908] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1}, {id:1002}]) [2025-01-08 00:51:47.828406] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:47.830099] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D0AA-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:47.831221] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:47.835522] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=85] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:47.836749] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=19] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:47.836837] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=18] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326304373137000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:47.838103] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=9] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:47.838560] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=447] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=108) [2025-01-08 00:51:47.846128] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.848880] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:47.850470] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=48] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:47.850590] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=46] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326305683208000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:47.850641] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=48] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326305683208000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326305683208000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:47.854250] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=21] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:47.859356] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326307859345000, start_id=1736326307859345000, end_id=1736326307859345001) [2025-01-08 00:51:47.859412] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=41] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:47.860504] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:47.860536] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-01-08 00:51:47.860546] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:47.860560] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:47.860742] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=179] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193811774196, flush_io_finished_ret_:0, flush_io_finished_round_:3794, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:47.880762] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185446-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:47.881131] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:47.881173] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-01-08 00:51:47.881224] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=49] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:47.881240] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:47.881253] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193811792773, flush_io_finished_ret_:0, flush_io_finished_round_:1966, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:47.881357] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=101] ====== traversal_flush timer task ====== [2025-01-08 00:51:47.881497] INFO [MDS] flush (mds_table_mgr.cpp:195) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385713-0-0] [lt=32] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326307827831000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319254996000, latest_log_ts:{val:1736326302607470001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.881622] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385713-0-0] [lt=119] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:47.887867] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:47.900704] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:47.901895] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=49] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:47.902011] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=41] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326305812845000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:47.902040] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326305812845000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326305812845000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:47.902096] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:47.902157] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.902276] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.902306] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.902329] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:47.902696] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:51:47.903302] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:47.908689] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C6-0-0] [lt=103][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193812822522, cache_obj->added_lc()=false, cache_obj->get_object_id()=13677, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:47.909136] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C7-0-0] [lt=22][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:47.910144] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=6239232) [2025-01-08 00:51:47.912597] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857C9-0-0] [lt=16][errcode=-4201] hashtable not init [2025-01-08 00:51:47.918859] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:47.946285] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:47.946325] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] ====== checkpoint timer task ====== [2025-01-08 00:51:47.947065] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=40] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:47.947108] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=42] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:47.947177] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326320087570000, latest_log_ts:{val:1736326303837002001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.947369] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=151] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:47.947392] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:47.947407] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326303837002001, v:0}) [2025-01-08 00:51:47.947415] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:47.947430] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:47.947449] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:47.947461] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:47.947480] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:47.947501] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:47.948191] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:47.948285] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=93] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:47.948325] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326302850005000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.948383] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=56] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:47.948394] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:47.948407] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:47.948420] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:47.948435] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2206509}, scn={val:1736326307358778004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:47.950642] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=27] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8790, offset=2496788}, end_cursor:ObLogCursor{file_id=1, log_id=8791, offset=2497085}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c54050, len:1772, log_data_len:297, seq:8790, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:47.951026] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326307358778004, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:47.951074] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:47.951110] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=35] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:47.952095] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:47.952148] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=52] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:47.952340] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326302846602000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.952549] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=206] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:47.952566] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:47.952581] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:47.952596] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:47.952609] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2240389}, scn={val:1736326307860790000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:47.953760] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:47.954300] INFO [PALF] pread (log_reader.cpp:130) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1286) [2025-01-08 00:51:47.954977] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8791, offset=2497085}, end_cursor:ObLogCursor{file_id=1, log_id=8792, offset=2497382}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c50050, len:1475, log_data_len:297, seq:8791, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:47.955035] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326307860790000, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:47.955059] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:47.955078] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:47.955924] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:47.955956] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=32] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:47.956035] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=17] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326302835077000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326304852832005, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:47.956137] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=99] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:47.956150] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:47.956163] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:47.956176] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:47.956258] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=77] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2227868}, scn={val:1736326307847404000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:47.956810] INFO [PALF] pread (log_reader.cpp:130) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=977) [2025-01-08 00:51:47.957711] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=26] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8792, offset=2497382}, end_cursor:ObLogCursor{file_id=1, log_id=8793, offset=2497679}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c58050, len:1178, log_data_len:297, seq:8792, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:47.957772] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326307847404000, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:47.957798] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:47.957818] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:47.957828] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:47.989258] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:48.007209] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=42] PNIO [ratelimit] time: 1736326308007204, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007684, add_bytes: 0 [2025-01-08 00:51:48.022536] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:48.023224] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58612][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=46] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:48.028117] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AE47-0-0] [lt=135] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.817210960388183594e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.518728175561641365e+02) [2025-01-08 00:51:48.028891] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:48.034627] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:51:48.046736] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.050178] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AC9-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18304, log_task={header:{begin_lsn:{lsn:2227868}, end_lsn:{lsn:2227990}, log_id:18304, min_scn:{val:1736326307847404001, v:0}, max_scn:{val:1736326307847404001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2227746}, prev_proposal_id:1, committed_end_lsn:{lsn:2227868}, data_checksum:940009244, accum_checksum:2179441060, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326307949979, freeze_ts:1736326307949979, submit_ts:1736326307950060, flushed_ts:1736326307953787, gen_to_freeze cost time:0, gen_to_submit cost time:81, submit_to_flush cost time:3727}, fs_cb_begin_ts=1736326308050094, log_life_time=100115) [2025-01-08 00:51:48.050206] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18214, miss_cnt:0, cache_read_size:2227990, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:48.093715] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=81] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:48.114500] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=82] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=89, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:48.114606] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:48.114681] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=68, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:48.114776] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:48.114930] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:324K/76:112236 disk:324K/76:112236 t1m2:40K/9:32492 t1m2g0:40K/9:32492 t1002m2:208K/51:100410 t1002m2g0:208K/51:100410 t1001m2:76K/16:194790 t1001m2g0:76K/16:194790 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:48.117217] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A958D-0-0] [lt=60] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:48.137561] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=74] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=3417, avg_group_log_size=379) [2025-01-08 00:51:48.146878] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=89][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.156619] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A958F-0-0] [lt=87] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:48.156650] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A958F-0-0] [lt=30] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22481, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:48.156660] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A958F-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=42, avg_log_submit_to_flush_time=1959, avg_log_submit_to_slide_time=22439) [2025-01-08 00:51:48.156792] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=52, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:48.156810] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=18] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=4, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:48.177454] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58601][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD92-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326308288967, keepalive_ts:-1}) [2025-01-08 00:51:48.177496] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58601][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD92-0-0] [lt=40] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326308288967, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:48.178827] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB618-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:48.178875] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB618-0-0] [lt=31] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326307847404003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326307847404003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:48.179397] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8793, offset=2497679}, end_cursor:ObLogCursor{file_id=1, log_id=8794, offset=2497984}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c5c050, len:881, log_data_len:305, seq:8793, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:48.179683] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6F-0-0] [lt=106] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:48.179716] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6F-0-0] [lt=20] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326307859345001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326307859345001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:48.184990] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326298176971, end_time=1736326308184989, dag_status={}) [2025-01-08 00:51:48.185015] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:48.185027] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:48.185035] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:48.185042] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:48.185049] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:48.185055] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:48.185062] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:48.185069] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:48.185075] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:48.185083] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185093] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185102] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185109] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185117] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185125] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185133] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185141] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185149] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185157] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185165] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185173] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185180] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185188] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185196] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185204] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185212] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185219] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185228] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185315] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=86] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185323] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185331] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185339] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185347] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185354] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185362] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185370] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185378] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185386] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185394] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185402] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185410] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185417] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185425] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185433] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185441] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185449] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185456] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185464] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185472] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185537] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=64] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185552] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185544] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E6F-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326308180226000, start_id=1736326308180226000, end_id=1736326308180226001) [2025-01-08 00:51:48.185560] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185568] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185576] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185584] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185650] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=65] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185659] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185667] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185675] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185682] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185690] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185698] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185731] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=32] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185740] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185748] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185756] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185764] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185772] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185779] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185787] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185795] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185803] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185811] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185818] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185826] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185834] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185842] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185850] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185857] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185865] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185873] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185881] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:48.185890] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:48.185898] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:48.185905] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:48.185910] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:48.185915] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:48.185920] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:48.185926] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:48.185931] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:48.185936] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:48.185941] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:48.185947] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:48.189684] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:48.192355] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=25] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8794, offset=2497984}, end_cursor:ObLogCursor{file_id=1, log_id=8795, offset=2498281}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c64050, len:576, log_data_len:297, seq:8794, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:48.193008] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6F-0-0] [lt=18] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:48.193069] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E6F-0-0] [lt=47] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326307859345002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326307859345002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:48.199354] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:149) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB618-0-0] [lt=28] transaction rpc statistics(total_rt=4230, total_process=19, avg_rt=211) [2025-01-08 00:51:48.202130] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=23] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:48.202796] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8795, offset=2498281}, end_cursor:ObLogCursor{file_id=1, log_id=8796, offset=2498578}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c60050, len:4375, log_data_len:297, seq:8795, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:48.204381] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8796, offset=2498578}, end_cursor:ObLogCursor{file_id=1, log_id=8797, offset=2498875}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c68050, len:4078, log_data_len:297, seq:8796, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:48.216030] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:48.229647] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:48.232856] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:48.233017] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58603][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=36] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:51:48.234802] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F852A7-0-0] [lt=36] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:48.249325] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.249943] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CBA-0-0] [lt=56][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:48.252095] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CBA-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:48.276361] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A482D-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:48.280632] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=60] ls blacklist refresh finish(cost_time=21495) [2025-01-08 00:51:48.294074] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:48.294402] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=54, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:48.297548] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EBE-0-0] [lt=43] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10571) [2025-01-08 00:51:48.297642] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EBE-0-0] [lt=30] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=23, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:48.297657] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EBE-0-0] [lt=15] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4379, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:48.298908] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58614][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:48.299000] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:51:48.314844] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:48.314883] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:48.317816] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3406, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:48.317848] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:48.317958] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=655, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:48.317993] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=27, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:48.319090] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.319142] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.342075] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:48.342129] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=46] temporary file module use no blocks [2025-01-08 00:51:48.342142] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:48.342157] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:48.342172] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193812255376, flush_io_finished_ret_:0, flush_io_finished_round_:1971, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:48.342875] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:48.343022] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:48.347742] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE91D-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326308346480000, start_id=1736326308346480000, end_id=1736326308346480001) [2025-01-08 00:51:48.349394] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.353844] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:48.361896] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=19] ts response task statistics(alloc_count=60770, free_count=30384) [2025-01-08 00:51:48.367851] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=54] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:51:48.368263] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=29] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:51:48.369997] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=631246848, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:51:48.370030] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] === Run print tenant memory usage task === [2025-01-08 00:51:48.370075] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=13] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,874,944 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 633,344,000 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,308,366,210 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 160,172,020 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,044,885,504 max_mem_memstore_can_get_now= 800,860,136 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,308,366,210 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 70,914,440 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 649,809,920 max_mem_memstore_can_get_now= 354,572,248 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,308,366,210 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 240,253,320 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,118,584,832 max_mem_memstore_can_get_now= 1,201,266,668 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:51:48.370473] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=10] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,657,314,304 total_hold= 3,856,662,528 used= 3,477,770,240 freelists_hold= 179,544,064 total_maps= 1,636 total_unmaps= 244 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,874,944 virtual_memory_used= 4,829,478,912 [CHUNK_MGR] 2 MB_CACHE: hold= 77,594,624 count= 37 pushes= 1,839,259 pops= 1,839,222 maps= 1,505 unmaps= 239 [CHUNK_MGR] 4 MB_CACHE: hold= 7,745,536 count= 3 pushes= 6,154 pops= 6,151 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,142 pops= 3,138 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:51:48.370500] INFO print (ob_malloc_time_monitor.cpp:37) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=12] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 2357, delta_count= 1, avg_cost_time= 2357 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 28597, delta_count= 1, avg_cost_time= 28597 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-01-08 00:51:48.374520] INFO [CLOG] update_upstream (ob_remote_location_adaptor.cpp:91) [50440][T1002_LogRessvr][T1002][YB42C0A89F89-00062B2D1BA8458E-0-0] [lt=17] primary tenant, just skip [2025-01-08 00:51:48.392328] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=30] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:48.392578] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=46] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:48.404047] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:48.406154] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:48.406416] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=39] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:48.410299] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:48.414848] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:48.414953] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=99] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:48.419500] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=72] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:48.426703] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD2-0-0] [lt=17] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:48.426763] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD2-0-0] [lt=56] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:48.430511] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:48.430653] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10571) [2025-01-08 00:51:48.433239] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=133] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326308092647000, v:0}, new_max_version={val:1736326308092647000, v:0}, last_min_version={val:1736326307893651000, v:0}, last_max_version={val:1736326307893651000, v:0}, total_time_us=6279, delta_us=340589) [2025-01-08 00:51:48.433273] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326308092647000, v:0}, min_version={val:1736326308092647000, v:0}, max_version={val:1736326308092647000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:48.435150] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2316, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:48.435296] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=129] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=108, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:48.435454] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=482] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=88, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:48.449507] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.458930] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.459366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.462844] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=77] gc stale ls task succ [2025-01-08 00:51:48.464544] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO [ratelimit] time: 1736326308464541, bytes: 882325809, bw: 0.299057 MB/s, add_ts: 1003687, add_bytes: 314740 [2025-01-08 00:51:48.466235] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4831-0-0] [lt=46] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:48.466295] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4831-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:48.495431] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=44] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9983, avg_group_log_size=623) [2025-01-08 00:51:48.495473] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9982) [2025-01-08 00:51:48.496669] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=114] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2381, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:48.496725] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=372, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:48.496839] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=23, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:48.499403] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EC0-0-0] [lt=69] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=242, avg_log_gen_to_submit_time=320, avg_log_submit_to_flush_time=2956, avg_log_submit_to_slide_time=4001) [2025-01-08 00:51:48.499599] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9983, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:48.517454] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:48.518343] INFO [LIB] stat (utility.h:1210) [58610][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8965-0-0] [lt=38] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:48.518369] INFO [LIB] stat (utility.h:1210) [58610][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8965-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5125, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:48.518379] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58610][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8965-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=119, avg_log_submit_to_flush_time=3533, avg_log_submit_to_slide_time=5006) [2025-01-08 00:51:48.518469] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:48.519366] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.519631] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A959A-0-0] [lt=28] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:48.519721] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A959A-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4952, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:48.519858] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:48.523895] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=24] [thread pool stat](pool_stat={pool:0x5572d527f900, last_check_time:1736326308523891, last_idle_time:15227232131, last_thread_count:4}, *pool={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15227232131, tenant_id:500}, qsize=-4, current_idle_time=15227232131) [2025-01-08 00:51:48.523933] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=37] [thread pool stat](pool_stat={pool:0x2ad8885abcf0, last_check_time:1736326308523933, last_idle_time:5295514446, last_thread_count:2}, *pool={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5295514446, tenant_id:500}, qsize=-1, current_idle_time=5295514446) [2025-01-08 00:51:48.523946] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] [thread pool stat](pool_stat={pool:0x2ad8885acaf0, last_check_time:1736326308523946, last_idle_time:3804726508, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885acaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3804726508, tenant_id:500}, qsize=-1, current_idle_time=3804726508) [2025-01-08 00:51:48.523957] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8881fc3b0, last_check_time:1736326308523956, last_idle_time:15155390615, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15155390615, tenant_id:1}, qsize=-4, current_idle_time=15155390615) [2025-01-08 00:51:48.523967] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8885ad1f0, last_check_time:1736326308523966, last_idle_time:3716730520, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885ad1f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3716730520, tenant_id:500}, qsize=-1, current_idle_time=3716730520) [2025-01-08 00:51:48.523976] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8c0eb0e70, last_check_time:1736326308523976, last_idle_time:7957765494, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7957765494, tenant_id:1002}, qsize=-4, current_idle_time=7957765494) [2025-01-08 00:51:48.523986] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8f90b0e70, last_check_time:1736326308523986, last_idle_time:7921036107, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7921036107, tenant_id:1001}, qsize=-4, current_idle_time=7921036107) [2025-01-08 00:51:48.530977] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C2-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:48.531034] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C2-0-0] [lt=31] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326308359531001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326308359531001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:48.538450] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:48.542064] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=138] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3417) [2025-01-08 00:51:48.542405] INFO [LIB] stat (utility.h:1210) [58601][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D0A-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=47, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:48.542426] INFO [LIB] stat (utility.h:1210) [58601][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D0A-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=3882, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:48.542437] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58601][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D0A-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=88, avg_log_submit_to_flush_time=2880, avg_log_submit_to_slide_time=3794) [2025-01-08 00:51:48.542658] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.543149] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3417, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:48.543137] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58608][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69A2-0-0] [lt=20] init point thread id with(&point=0x5572d50e6540, idx_=1264, point=[thread id=58608, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58608) [2025-01-08 00:51:48.544485] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.549633] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.551263] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58585][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AD3-0-0] [lt=62] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:48.567292] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58615][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:48.568077] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:48.572644] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=37][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:48.572680] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=16][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:48.573359] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=26][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:48.573408] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2482, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:48.573488] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=71] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=100, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:48.573598] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=107] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=70, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:48.589439] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6CC-0-0] [lt=37][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193813503271, cache_obj->added_lc()=false, cache_obj->get_object_id()=13678, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:48.592812] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:48.617979] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2817, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:48.618122] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=714, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:48.618199] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:48.631240] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:48.632804] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=69, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:48.632855] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3417) [2025-01-08 00:51:48.634861] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3418) [2025-01-08 00:51:48.638097] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=69] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:48.649679] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.650858] INFO [LIB] stat (utility.h:1210) [58600][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AD5-0-0] [lt=42] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:48.650915] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58600][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AD5-0-0] [lt=45] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:48.652604] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1655, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:48.652633] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:48.652653] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=357, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:48.652732] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:48.671050] INFO [LIB] stat (utility.h:1210) [58596][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4835-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=38, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:48.671084] INFO [LIB] stat (utility.h:1210) [58596][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4835-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21899, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:48.671096] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58596][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4835-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=2676, avg_log_submit_to_slide_time=21878) [2025-01-08 00:51:48.671110] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58596][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4835-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18413, log_task={header:{begin_lsn:{lsn:2241121}, end_lsn:{lsn:2241243}, log_id:18413, min_scn:{val:1736326308359531002, v:0}, max_scn:{val:1736326308359531002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2240999}, prev_proposal_id:1, committed_end_lsn:{lsn:2241121}, data_checksum:1715074492, accum_checksum:2305503378, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326308567139, freeze_ts:1736326308567139, submit_ts:1736326308567152, flushed_ts:1736326308573553, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:6401}, fs_cb_begin_ts=1736326308670901, log_life_time=103762) [2025-01-08 00:51:48.671750] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:48.671785] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18321, miss_cnt:0, cache_read_size:2241243, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:48.673332] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:48.706482] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708516F-0-0] [lt=143] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:48.725092] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=133] ====== tenant freeze timer task ====== [2025-01-08 00:51:48.738433] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=33] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:48.744105] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:48.749820] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:48.749857] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.754025] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58546][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB5B-0-0] [lt=32] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:48.754060] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58546][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB5B-0-0] [lt=28] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:48.755522] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:62, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2040528697, recv_hp_rpc_cnt:5236, recv_np_rpc_cnt:104356, recv_lp_rpc_cnt:0, recv_mysql_cnt:9408, recv_task_cnt:3764, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10880462, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=116849 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1851, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:94426, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736326308232853}{group_id:10, queue_size:0, recv_req_cnt:11254, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2100, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326289381035}{group_id:2, queue_size:0, recv_req_cnt:30266, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11944, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:4998, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:26908, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=287 pcode=0x1502:cnt=208 pcode=0x717:cnt=185 pcode=0x710:cnt=66 pcode=0x150b:cnt=60, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:51:48.756330] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=806] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:16048466, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8240, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3600197, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=20, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:51:48.757068] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=736] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:15609964, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3586740, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:51:48.757728] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=658] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:23, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:887097586, recv_hp_rpc_cnt:3776, recv_np_rpc_cnt:38377, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:787760, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=66151 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:68589, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:6, nesting_worker_cnt:0, token_change:1736326308297280}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2214, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326279406832}{group_id:2, queue_size:0, recv_req_cnt:15661, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18780, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=308 pcode=0x51c:cnt=244 pcode=0x717:cnt=178 pcode=0x710:cnt=99 pcode=0x150b:cnt=60, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:51:48.758497] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=767] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:59, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9787577569, recv_hp_rpc_cnt:77, recv_np_rpc_cnt:35364, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2979320, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=25280 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:0, recv_req_cnt:7103, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326269316514}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:152099, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:8, nesting_worker_cnt:0, token_change:1736326308566912}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326302288855}{group_id:19, queue_size:0, recv_req_cnt:412, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326264650105}{group_id:2, queue_size:0, recv_req_cnt:37576, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:11235, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=400 pcode=0x1501:cnt=201 pcode=0x1516:cnt=195 pcode=0x717:cnt=190 pcode=0x150b:cnt=120, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:51:48.775227] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=114] clear dtl interm result cost(us)(clear_cost=3329, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:48.775334] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=55] clear dtl interm result cost(us)(clear_cost=13, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:48.775364] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43535187}, readable_log_tail:{lsn:43535187}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23569581, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:48.775447] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=82] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:48.775625] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=105] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:48.793503] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:48.793973] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1736326308347846004, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3, epoch_tstamp:1736326308793911}, version_delta=446126) [2025-01-08 00:51:48.807406] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=16] BlockGCTimerTask success(ret=0, cost_time_us=14, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:48.811097] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=51] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:48.811174] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44639, [delay/us]:prepare:6, schedule:334, submit:866, rt:1422, total:2630) [2025-01-08 00:51:48.811250] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:48.811336] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44636, [delay/us]: prepare:6, schedule:334, submit:866, device:1422, total:2630) [2025-01-08 00:51:48.811351] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:48.811390] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:14, bw:64925, [delay/us]:prepare:7, schedule:106, submit:737, rt:1178, total:2030) [2025-01-08 00:51:48.811427] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:48.811470] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:14, avg_bw:64922, [delay/us]: prepare:7, schedule:106, submit:737, device:1178, total:2030) [2025-01-08 00:51:48.811480] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:48.811503] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:7, bw:32471, [delay/us]:prepare:5, schedule:28, submit:0, rt:1490, total:1525) [2025-01-08 00:51:48.811531] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:170475, [delay/us]:prepare:9, schedule:126, submit:512, rt:1511, total:2160) [2025-01-08 00:51:48.811565] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12176, [delay/us]:prepare:2, schedule:109, submit:139, rt:250, total:500) [2025-01-08 00:51:48.811576] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:48.811607] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4681, avg_iops:7, avg_bw:32470, [delay/us]: prepare:5, schedule:28, submit:0, device:1490, total:1525) [2025-01-08 00:51:48.811621] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12176, [delay/us]: prepare:2, schedule:109, submit:139, device:250, total:500) [2025-01-08 00:51:48.811651] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:170464, [delay/us]: prepare:9, schedule:126, submit:512, device:1511, total:2160) [2025-01-08 00:51:48.811664] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:48.811724] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4058, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1521, total:1531) [2025-01-08 00:51:48.811775] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:48.811809] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4058, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1521, total:1531) [2025-01-08 00:51:48.811826] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:48.811870] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:48.811880] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:48.811888] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:48.811895] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=243kB/s, net_out=228kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:48.813731] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC413-0-0] [lt=63][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:48.818513] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:48.818556] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:48.819514] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0AD-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:48.822394] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=17] gc stale ls task succ [2025-01-08 00:51:48.822516] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:48.822727] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=30] gc stale ls task succ [2025-01-08 00:51:48.824313] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87157-0-0] [lt=11][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:48.824385] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:48.824398] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:48.828332] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1002}, {id:1001}]) [2025-01-08 00:51:48.831291] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:48.831713] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:48.832783] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43535775}, readable_log_tail:{lsn:43535775}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23568993, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:48.849961] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.854449] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=28] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:48.861439] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:48.861669] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=222] temporary file module use no blocks [2025-01-08 00:51:48.861683] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:48.861697] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:48.861710] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193812774618, flush_io_finished_ret_:0, flush_io_finished_round_:3795, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:48.862612] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326308862589], gts:1736326308862403000, latest_srr:[mts=1736326308862589]}) [2025-01-08 00:51:48.862641] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:48.862663] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326308862653], gts:1736326308862403000, latest_srr:[mts=1736326308862653]}) [2025-01-08 00:51:48.862938] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326308862925], gts:1736326308862403000, latest_srr:[mts=1736326308862925]}) [2025-01-08 00:51:48.882961] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185447-0-0] [lt=88][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:48.883345] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:48.883442] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=89] temporary file module use no blocks [2025-01-08 00:51:48.883454] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:48.883473] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:48.883486] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193812795123, flush_io_finished_ret_:0, flush_io_finished_round_:1967, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:48.892517] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=43] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:48.897347] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=74] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.897403] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.897431] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.897454] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:48.904419] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:48.906360] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:48.906790] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=127] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:48.912459] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:48.916862] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=43] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:48.920297] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=118] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:48.950133] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:48.963790] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58609][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF33-0-0] [lt=49] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326308976033, keepalive_ts:-1}) [2025-01-08 00:51:48.963854] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58609][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF33-0-0] [lt=61] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326308976033, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:48.968864] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE9-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:48.968898] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CE9-0-0] [lt=26] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=36) [2025-01-08 00:51:48.977915] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=35] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:48.994173] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:49.014823] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=60] PNIO [ratelimit] time: 1736326309014820, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-01-08 00:51:49.030337] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AE56-0-0] [lt=106] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.817210960388183594e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.518728175561641365e+02) [2025-01-08 00:51:49.032531] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:49.034629] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=106] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1258280, clean_num=31457) [2025-01-08 00:51:49.050270] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=112][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.054260] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18225, miss_cnt:0, cache_read_size:2229332, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:49.063608] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2CD-0-0] [lt=66] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326309076749, keepalive_ts:-1}) [2025-01-08 00:51:49.063649] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2CD-0-0] [lt=39] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326309076749, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:49.066157] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58615][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2CE-0-0] [lt=78] init point thread id with(&point=0x5572d50e6700, idx_=1271, point=[thread id=58615, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58615) [2025-01-08 00:51:49.066267] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58615][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2CE-0-0] [lt=105] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326309076923, keepalive_ts:-1}) [2025-01-08 00:51:49.066283] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58615][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2CE-0-0] [lt=15] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326309076923, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:49.096777] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:49.117873] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:49.117927] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:49.118124] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:49.118289] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=152] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:49.118508] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:296K/68:194918 disk:296K/68:194918 t1m2:44K/10:321114 t1m2g0:44K/10:321114 t1002m2:188K/44:162473 t1002m2g0:188K/44:162473 t1001m2:64K/14:206750 t1001m2g0:64K/14:206750 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:49.120262] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:49.121611] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326305096115, last_first_register_time_us=1736324449183670, register_time_us=1736326305096115, curr_time_us=1736326309121607) [2025-01-08 00:51:49.121687] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326309121643, last_submit_register_req_time_us=1736326309121643, register_parent_reason=1) [2025-01-08 00:51:49.121707] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:49.139198] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] no need to update min/max version(new_version={val:1736326308740603000, v:0}, min_version={val:1736326308740603000, v:0}, max_version={val:1736326308740603000, v:0}) [2025-01-08 00:51:49.139328] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3540, avg_group_log_size=354) [2025-01-08 00:51:49.150397] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.166836] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF69-0-0] [lt=16] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326309278458, keepalive_ts:-1}) [2025-01-08 00:51:49.166871] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF69-0-0] [lt=33] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326309278458, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:49.169340] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=64] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=73, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:49.169368] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:49.176719] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58601][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FC-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326309190496, keepalive_ts:-1}) [2025-01-08 00:51:49.176770] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58601][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FC-0-0] [lt=49] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326309190496, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:49.185119] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326299177044, end_time=1736326309185116, dag_status={}) [2025-01-08 00:51:49.185154] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=31] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:49.185265] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=110] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:49.185274] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:49.185282] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:49.185289] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:49.185296] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:49.185303] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:49.185310] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:49.185317] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:49.185324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185335] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185343] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185352] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185360] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185409] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=48] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185418] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185426] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185434] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185442] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185450] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185458] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185466] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185474] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185482] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185490] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185498] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185506] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185514] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185560] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=45] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185569] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185577] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185585] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185593] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185601] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185609] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185617] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185625] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185633] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185640] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185649] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185656] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185664] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185672] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185680] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185688] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=32] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185730] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185738] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185746] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185754] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185762] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185778] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185786] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185794] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185802] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185810] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185817] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185825] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185833] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185841] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185849] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185857] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185864] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185873] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185880] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185888] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185896] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185904] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185912] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185919] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185927] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185935] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185943] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185951] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.185959] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.186061] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=101] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.186076] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.186086] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.186095] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.186104] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.186112] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:49.186122] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:49.186130] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:49.186138] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:49.186143] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:49.186149] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:49.186154] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:49.186159] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:49.186164] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:49.186169] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:49.186174] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:49.186181] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:49.194508] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-01-08 00:51:49.194551] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=37] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:49.195888] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:49.208863] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8797, offset=2498875}, end_cursor:ObLogCursor{file_id=1, log_id=8798, offset=2499180}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c6c050, len:3781, log_data_len:305, seq:8797, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:49.209382] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E70-0-0] [lt=77] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:49.209429] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E70-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326308862403002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326308862403002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:49.217750] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB619-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:49.217805] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB619-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326308847457003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326308847457003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:49.219279] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=24] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:49.220722] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=73] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:49.227939] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=28] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:49.234492] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=948] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:49.235869] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:49.235943] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=38] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8798, offset=2499180}, end_cursor:ObLogCursor{file_id=1, log_id=8799, offset=2499477}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c74050, len:3476, log_data_len:297, seq:8798, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:49.237157] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58576][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=11] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:51:49.237530] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E70-0-0] [lt=78] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:49.237592] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E70-0-0] [lt=40] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326308862403002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326308862403002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:49.241625] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] no need to update min/max version(new_version={val:1736326308740603000, v:0}, min_version={val:1736326308740603000, v:0}, max_version={val:1736326308740603000, v:0}) [2025-01-08 00:51:49.243468] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=19] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:49.250538] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.250990] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8799, offset=2499477}, end_cursor:ObLogCursor{file_id=1, log_id=8800, offset=2499774}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c70050, len:3179, log_data_len:297, seq:8799, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:49.261909] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CBB-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:49.263609] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2229454}], scn_array=[cnt:1, {val:1736326308847457004, v:0}], curr_size=122, accum_size=732, time_cost=12089) [2025-01-08 00:51:49.263711] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Slow write(duration=12562, write_len=4096) [2025-01-08 00:51:49.263736] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=23] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8800, offset=2499774}, end_cursor:ObLogCursor{file_id=1, log_id=8801, offset=2500071}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c78050, len:2882, log_data_len:297, seq:8800, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:49.263955] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CBB-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:49.286651] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=38] ====== check clog disk timer task ====== [2025-01-08 00:51:49.286682] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:49.294252] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:49.298517] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=66, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:49.298822] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:49.299543] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58614][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=27] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:49.300346] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=8685) [2025-01-08 00:51:49.301755] INFO [LIB] stat (utility.h:1210) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B64-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=24, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:49.301778] INFO [LIB] stat (utility.h:1210) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B64-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4359, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:49.318853] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:49.318894] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:49.319240] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.319296] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.320716] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4078, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:49.320745] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:49.320764] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=171, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:49.320888] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=46, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:49.324585] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5296311978, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:49.324633] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:49.324648] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad90857ded0) [2025-01-08 00:51:49.324657] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5296311978, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:51:49.342408] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=199] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:49.342452] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=36] temporary file module use no blocks [2025-01-08 00:51:49.342464] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:49.342478] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:49.342492] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193813256218, flush_io_finished_ret_:0, flush_io_finished_round_:1972, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:49.347943] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=34] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:49.350960] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=68] get number(ret=0, service_type_=0, range=1, base_id=1736326309350305000, start_id=1736326309350305001, end_id=1736326309350305002) [2025-01-08 00:51:49.351127] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=109] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:49.351164] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.352097] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AE3-0-0] [lt=105] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:49.352138] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AE3-0-0] [lt=40] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12256, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:49.352152] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AE3-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=2639, avg_log_submit_to_slide_time=12237) [2025-01-08 00:51:49.352166] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AE3-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18317, log_task={header:{begin_lsn:{lsn:2229454}, end_lsn:{lsn:2229576}, log_id:18317, min_scn:{val:1736326308847457004, v:0}, max_scn:{val:1736326308847457004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2229332}, prev_proposal_id:1, committed_end_lsn:{lsn:2229454}, data_checksum:940009244, accum_checksum:864895150, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326309251378, freeze_ts:1736326309251378, submit_ts:1736326309251388, flushed_ts:1736326309263664, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:12276}, fs_cb_begin_ts=1736326309352084, log_life_time=100706) [2025-01-08 00:51:49.358729] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:49.363541] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=504, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:49.369003] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4843-0-0] [lt=138] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:49.387041] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=46] current monitor number(seq_=-1) [2025-01-08 00:51:49.387659] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC3-0-0] [lt=26] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:49.387705] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC3-0-0] [lt=37] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:49.392776] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:49.396404] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=95] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:49.405725] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D4E-0-0] [lt=46] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:49.406368] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=91] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:49.406542] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=31] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:49.408344] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=179] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:49.408472] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:49.408497] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:49.408509] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:49.408520] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:49.411896] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:49.412759] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=858] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:49.413523] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:51:49.413783] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:49.418930] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:49.418963] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:49.419033] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22] ====== tenant freeze timer task ====== [2025-01-08 00:51:49.420784] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:49.425526] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB67-0-0] [lt=29] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:49.425571] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB67-0-0] [lt=36] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:49.429465] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD3-0-0] [lt=47] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:49.429503] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD3-0-0] [lt=35] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:49.433535] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=60] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9274) [2025-01-08 00:51:49.435484] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326309226495001, v:0}, new_max_version={val:1736326309226495001, v:0}, last_min_version={val:1736326309029441000, v:0}, last_max_version={val:1736326309029441000, v:0}, total_time_us=6498, delta_us=208985) [2025-01-08 00:51:49.435534] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326309226495001, v:0}, min_version={val:1736326309226495001, v:0}, max_version={val:1736326309226495001, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:49.436450] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:49.451324] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.458653] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.459020] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.462955] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=42] gc stale ls task succ [2025-01-08 00:51:49.463618] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-01-08 00:51:49.463646] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=20] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70914440, tenant_id=1001) [2025-01-08 00:51:49.463720] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:49.466525] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4845-0-0] [lt=40] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:49.466664] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4845-0-0] [lt=124] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:49.468667] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=95] PNIO [ratelimit] time: 1736326309468664, bytes: 882598354, bw: 0.258852 MB/s, add_ts: 1004123, add_bytes: 272545 [2025-01-08 00:51:49.496584] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326308847457005, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=649114, in_cluster_service=true, cluster_version={val:1736326308847457002, v:0}, min_cluster_version={val:1736326308847457002, v:0}, max_cluster_version={val:1736326308847457002, v:0}, get_cluster_version_err=0, cluster_version_delta=649114, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18627, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=537414098, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:49.499452] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=192] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=9274, avg_group_log_size=662) [2025-01-08 00:51:49.499513] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=54] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9274) [2025-01-08 00:51:49.501368] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=2441, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:49.501415] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=246, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:49.501507] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:49.503319] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B66-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=132, avg_log_gen_to_submit_time=212, avg_log_submit_to_flush_time=2852, avg_log_submit_to_slide_time=4101) [2025-01-08 00:51:49.503528] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9274, hit_cnt=14, read_cnt=14, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:49.513251] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=42] LogLoopThread round_cost_time(us)(round_cost_time=18) [2025-01-08 00:51:49.513357] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=47] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:45001, max_lsn:84434107, max_scn:{val:1736326309341577001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84434107}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42491067}}, last_submit_lsn:{lsn:84433521}, last_submit_end_lsn:{lsn:84434107}, last_submit_log_id:45001, last_submit_log_pid:1, max_flushed_lsn:{lsn:84433521}, max_flushed_end_lsn:{lsn:84434107}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84434107}, last_slide_log_id:45001, last_slide_scn:{val:1736326309341577001, v:0}, last_slide_lsn:{lsn:84433521}, last_slide_end_lsn:{lsn:84434107}, last_slide_log_pid:1, last_slide_log_accum_checksum:1761676169, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:45002, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84434107}, readable_log_tail:{lsn:84434107}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49775429, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:51:49.513713] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=349] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326309346825, last_flushed_end_lsn:{lsn:84434107}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326309350780, last_flushed_end_lsn:{lsn:84434107}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326309353261, last_flushed_end_lsn:{lsn:84434107}}]) [2025-01-08 00:51:49.513737] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=45002, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326309346825, last_flushed_end_lsn:{lsn:84434107}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326309350780, last_flushed_end_lsn:{lsn:84434107}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326309353261, last_flushed_end_lsn:{lsn:84434107}}]) [2025-01-08 00:51:49.519362] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.520934] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=93] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2212435, this={dio_aligned_buf:{buf_write_offset:717, buf_padding_size:3379, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:23007, truncate_used_ts:5204}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2211840, buf_len=122, total_write_size=12294, total_write_size_after_dio=425984, ob_pwrite_used_ts=320449, count=101, each_pwrite_cost=3172) [2025-01-08 00:51:49.521851] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:49.523022] INFO [LIB] stat (utility.h:1210) [58610][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95B7-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:49.523049] INFO [LIB] stat (utility.h:1210) [58610][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95B7-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5264, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:49.523059] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58610][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95B7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=85, avg_log_submit_to_flush_time=4295, avg_log_submit_to_slide_time=5178) [2025-01-08 00:51:49.523116] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] LogLoopThread round_cost_time(us)(round_cost_time=34) [2025-01-08 00:51:49.523158] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=66] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:49.524287] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8984-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:49.524401] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8984-0-0] [lt=112] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5375, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:49.525918] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:49.531535] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=171] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:49.540427] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:49.541660] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.544669] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58608][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D14-0-0] [lt=119] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3076) [2025-01-08 00:51:49.544832] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.544884] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2138, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:49.544910] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=390, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:49.545011] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:49.545032] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4409, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:49.545045] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=72, avg_log_gen_to_submit_time=204, avg_log_submit_to_flush_time=2790, avg_log_submit_to_slide_time=4205) [2025-01-08 00:51:49.545073] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=79, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:49.545119] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=91] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3076, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:49.551438] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=109][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.555369] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58600][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AE7-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:49.561197] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C3-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:49.561268] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C3-0-0] [lt=51] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326309361852001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326309361852001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:49.575894] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:49.576017] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58610][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=11] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:49.582540] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAECF-0-0] [lt=81] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:49.582655] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAECF-0-0] [lt=34] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:49.586492] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:51:49.596697] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:49.608325] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6CE-0-0] [lt=48][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193814522156, cache_obj->added_lc()=false, cache_obj->get_object_id()=13679, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:49.621795] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4040, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:49.621949] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=156, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:49.622115] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=34, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:49.636741] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get_max_applied_scn(scn={val:1736326309495692000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43543872}, palf_committed_end_scn_:{val:1736326309628490000, v:0}, last_check_scn_:{val:1736326309628490000, v:0}, max_applied_cb_scn_:{val:1736326309495692000, v:0}}) [2025-01-08 00:51:49.637173] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:49.640392] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:49.640438] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=43] try gc retain ctx [2025-01-08 00:51:49.640449] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-01-08 00:51:49.640474] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1736326309543479000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84434815}, palf_committed_end_scn_:{val:1736326309543479000, v:0}, last_check_scn_:{val:1736326309543479000, v:0}, max_applied_cb_scn_:{val:1736326309543479000, v:0}}) [2025-01-08 00:51:49.640494] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84434815}, readable_log_tail:{lsn:84434815}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49774721, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:49.640648] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=149] finish update min start scn(max_decided_scn={val:1736326309543479000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326304415504000, v:0}, keep_alive_scn_:{val:1736326304616286001, v:0}, update_ts_:1736326309639660}) [2025-01-08 00:51:49.641383] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319254996000, latest_log_ts:{val:1736326302607470001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:49.641861] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=345] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=71, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:49.642014] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3662) [2025-01-08 00:51:49.643470] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3189) [2025-01-08 00:51:49.649050] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=126] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:49.649131] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=80] try gc retain ctx [2025-01-08 00:51:49.649141] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-01-08 00:51:49.649169] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326309628490000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326299572894001, v:0}, keep_alive_scn_:{val:1736326304600320001, v:0}, update_ts_:1736326309648834}) [2025-01-08 00:51:49.649983] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=120] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319284252000, latest_log_ts:{val:1736326302668823001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:49.651598] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.652275] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AE9-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:49.652403] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AE9-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:49.653841] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2889, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:49.653883] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:49.653904] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=140, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:49.653978] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:49.668312] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=56] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1944, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:49.668381] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=83, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:49.668452] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=90, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:49.702679] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58618][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:49.703544] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:51:49.714627] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=31] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:49.734887] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=41] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:49.734909] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:49.735032] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498F-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:49.735052] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8498F-0-0] [lt=17] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:49.742934] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] no need to update min/max version(new_version={val:1736326309341577001, v:0}, min_version={val:1736326309341577001, v:0}, max_version={val:1736326309341577001, v:0}) [2025-01-08 00:51:49.751788] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.767877] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:49.769736] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC898B-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=19, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:49.769764] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC898B-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3710, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:49.769775] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC898B-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=35, avg_log_submit_to_flush_time=2006, avg_log_submit_to_slide_time=3675) [2025-01-08 00:51:49.770317] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1464, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:49.770355] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18333, miss_cnt:0, cache_read_size:2242707, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:49.784635] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] success to update min reserved snapshot(reserved_snapshot=1736324505683208000, duration=1800, snapshot_gc_ts=1736326305683208000) [2025-01-08 00:51:49.788032] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=50] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:49.790075] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=59] get number(ret=0, service_type_=0, range=1, base_id=1736326309789748000, start_id=1736326309789748000, end_id=1736326309789748001) [2025-01-08 00:51:49.790277] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18730684, this={dio_aligned_buf:{buf_write_offset:3884, buf_padding_size:212, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:32930, truncate_used_ts:6769}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=18726912, buf_len=112, total_write_size=14974, total_write_size_after_dio=421888, ob_pwrite_used_ts=369667, count=100, each_pwrite_cost=3696) [2025-01-08 00:51:49.790971] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=33] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324505683208000) [2025-01-08 00:51:49.794829] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=24] submit reserved snapshot log success(ls_id={id:1001}, new_snapshot_version=1736324505683208000) [2025-01-08 00:51:49.794860] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=26] update reserved snapshot finished(cost_ts=10231, reserved_snapshot=1736324505683208000) [2025-01-08 00:51:49.796909] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=100] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:49.817760] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:49.817796] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:49.818023] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:49.818144] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44744, [delay/us]:prepare:7, schedule:115, submit:224, rt:1167, total:1514) [2025-01-08 00:51:49.818163] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:49.818452] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=270] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44737, [delay/us]: prepare:7, schedule:115, submit:224, device:1167, total:1514) [2025-01-08 00:51:49.818515] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:49.818590] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:15, bw:69134, [delay/us]:prepare:12, schedule:315, submit:250, rt:1826, total:2404) [2025-01-08 00:51:49.818603] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:49.818694] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=74] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:15, avg_bw:69132, [delay/us]: prepare:12, schedule:315, submit:250, device:1826, total:2404) [2025-01-08 00:51:49.818754] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:49.818801] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16265, [delay/us]:prepare:6, schedule:1247, submit:1, rt:4444, total:5698) [2025-01-08 00:51:49.818838] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4286, iops:43, bw:182984, [delay/us]:prepare:11, schedule:188, submit:601, rt:2011, total:2813) [2025-01-08 00:51:49.818914] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:49.819084] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=163] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16262, [delay/us]: prepare:6, schedule:1247, submit:1, device:4444, total:5698) [2025-01-08 00:51:49.819102] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4287, avg_iops:43, avg_bw:182956, [delay/us]: prepare:11, schedule:188, submit:601, device:2011, total:2813) [2025-01-08 00:51:49.819112] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:49.819133] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4065, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:358, total:368) [2025-01-08 00:51:49.819163] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:49.819294] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=124] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4065, [delay/us]: prepare:0, schedule:0, submit:600000000, device:358, total:368) [2025-01-08 00:51:49.819314] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:49.819332] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:49.819341] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:49.819349] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:49.819357] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=243kB/s, net_out=225kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:49.822795] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC416-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:49.822995] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=17] gc stale ls task succ [2025-01-08 00:51:49.823083] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=32] ====== check clog disk timer task ====== [2025-01-08 00:51:49.823107] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:49.823127] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=182] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:49.823219] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=12] gc stale ls task succ [2025-01-08 00:51:49.825807] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8715D-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:49.825855] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:49.825866] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:49.828684] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=91] start do ls ha handler(ls_id_array_=[{id:1}, {id:1003}, {id:1001}, {id:1002}]) [2025-01-08 00:51:49.828915] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0B0-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:49.831747] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=37] ====== checkpoint timer task ====== [2025-01-08 00:51:49.831746] INFO pn_ratelimit (group.c:653) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:49.831819] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=311] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:49.832763] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=22] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:1, active_tx_count:1, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:49.832928] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=164] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1736326309830872000, v:0}) [2025-01-08 00:51:49.833067] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=43] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319284252000, latest_log_ts:{val:1736326302668823001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:49.833294] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=222] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:49.833327] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=27] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:49.833359] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=29] get rec log scn(service_type_=0, rec_log_ts={val:1736326302668823001, v:0}) [2025-01-08 00:51:49.833367] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:49.833374] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:49.833386] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:49.833429] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=38] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:49.833443] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:49.833469] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:49.833492] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:49.833502] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:49.837383] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=7] finish to resize log external storage handler(current_ts=1736326309837379, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:49.837442] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=57] snapshot_traffic_info once [2025-01-08 00:51:49.837765] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=121] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:49.845502] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1736326309844896000, start_id=1736326309844896000, end_id=1736326309844896001) [2025-01-08 00:51:49.846949] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=27] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=649809920, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=423931904, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:49.847052] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=65] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:49.847074] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2881, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:49.847215] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1116487680, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2104737792, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:49.847291] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=41] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:49.847378] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=85] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:49.851945] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.854639] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=40] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:49.862240] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:49.862282] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=34] temporary file module use no blocks [2025-01-08 00:51:49.862293] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:49.862320] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:49.862334] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193813775580, flush_io_finished_ret_:0, flush_io_finished_round_:3796, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:49.864030] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=46] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:49.868826] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385715-0-0] [lt=10] MediumCheckTask(cost_ts=46) [2025-01-08 00:51:49.885151] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185448-0-0] [lt=88][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:49.885475] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=43] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:49.885501] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-01-08 00:51:49.885510] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:49.885522] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:49.885555] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=32] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193813797374, flush_io_finished_ret_:0, flush_io_finished_round_:1968, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:49.893013] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=47] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:49.896795] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:49.898972] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857D9-0-0] [lt=64] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=108) [2025-01-08 00:51:49.899329] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=48] get number(ret=0, service_type_=0, range=1, base_id=1736326309898649000, start_id=1736326309898649000, end_id=1736326309898649001) [2025-01-08 00:51:49.902808] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.902910] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.903017] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.903044] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:49.907178] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:49.908981] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:49.912821] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=39] ====== checkpoint timer task ====== [2025-01-08 00:51:49.913946] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=40] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:49.913995] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=48] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:49.914124] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319254996000, latest_log_ts:{val:1736326302607470001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:49.914285] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=157] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319254996000, latest_log_ts:{val:1736326302607470001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:49.914394] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=55] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:49.914415] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=17] get rec log scn(service_type_=0, rec_log_ts={val:1736326302607470001, v:0}) [2025-01-08 00:51:49.914423] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:49.914431] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:49.914445] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:49.914466] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:49.914479] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:49.914500] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:49.914522] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:49.914596] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=71] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:49.914688] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:49.916479] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:49.921935] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:49.931302] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=127] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326309830872000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=100421, in_cluster_service=true, cluster_version={val:1736326309628490000, v:0}, min_cluster_version={val:1736326309628490000, v:0}, max_cluster_version={val:1736326309628490000, v:0}, get_cluster_version_err=0, cluster_version_delta=302803, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=19064, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=655568076, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:49.935965] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AB2-0-0] [lt=98] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:49.939273] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=28] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:49.940980] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1736326309938244000, start_id=1736326309938244000, end_id=1736326309938244001) [2025-01-08 00:51:49.943427] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=106] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:49.944363] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=21] [APPLY STAT CB APPEND COST TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=127, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:49.944459] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=95] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=4437, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:49.944472] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=13] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:49.944480] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=-927, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:49.944487] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=3637, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:49.952070] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:49.960074] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=27] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12799463424, fast_delete_log_mode=false) [2025-01-08 00:51:49.979327] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=113] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:49.998044] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=67] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1478432, replace_num=15728) [2025-01-08 00:51:49.998094] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:50.020777] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BB9-0-0] [lt=27] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:50.022439] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=69] PNIO [ratelimit] time: 1736326310022405, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007585, add_bytes: 0 [2025-01-08 00:51:50.033326] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=27] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:50.038423] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:50.045341] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D1969101F-0-0] [lt=52] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.784745297431945801e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=892, Throttle Trigger(MB)=535, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.806947799604412097e+01) [2025-01-08 00:51:50.052175] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.052483] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:50.054065] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=61] succ to submit log task to replay service(to_submit_lsn={lsn:2230477}, to_submit_scn={val:1736326309849297002, v:0}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326310053920, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2230543}, next_to_submit_scn_:{val:1736326309849297003, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2230421}, end_lsn:{lsn:2230543}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2230543}, readable_log_tail:{lsn:2230543}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64874225, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326309849297002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326309849297002, v:0}, data_checksum:2124447694, flag:37998}}, init_mode_version:0, accumulate_checksum:2643858968, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326309849297002, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18235, miss_cnt:0, cache_read_size:2230543, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:50.102549] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:50.120425] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=100, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:50.120641] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=198] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:50.120716] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=65, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:50.120741] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:50.120904] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/73:203768 disk:308K/73:203768 t1m2:56K/13:124341 t1m2g0:56K/13:124341 t1002m2:180K/44:235728 t1002m2g0:180K/44:235728 t1001m2:72K/16:173302 t1001m2g0:72K/16:173302 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:50.122782] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1210) [2025-01-08 00:51:50.124805] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326306107147, last_first_register_time_us=1736324454612308, register_time_us=1736326306107147, curr_time_us=1736326310124803) [2025-01-08 00:51:50.125022] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=183] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326310124991, last_submit_register_req_time_us=1736326310124991, register_parent_reason=1) [2025-01-08 00:51:50.125079] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=54] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:50.152231] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=89][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.155407] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18236, miss_cnt:0, cache_read_size:2230665, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:50.169721] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4853-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18428, log_task={header:{begin_lsn:{lsn:2242951}, end_lsn:{lsn:2243073}, log_id:18428, min_scn:{val:1736326309861406002, v:0}, max_scn:{val:1736326309861406002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2242829}, prev_proposal_id:1, committed_end_lsn:{lsn:2242951}, data_checksum:1715074492, accum_checksum:1204471466, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326310066924, freeze_ts:1736326310066924, submit_ts:1736326310066939, flushed_ts:1736326310068333, gen_to_freeze cost time:0, gen_to_submit cost time:15, submit_to_flush cost time:1394}, fs_cb_begin_ts=1736326310169707, log_life_time=102783) [2025-01-08 00:51:50.169974] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=52] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=57, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:50.170017] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:50.171537] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF6A-0-0] [lt=119] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326310280398, keepalive_ts:-1}) [2025-01-08 00:51:50.171754] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF6A-0-0] [lt=214] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326310280398, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:50.183432] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A6D-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:50.198302] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:50.219526] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A6E-0-0] [lt=38] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:50.223467] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1633) [2025-01-08 00:51:50.232537] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A6F-0-0] [lt=38] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:50.237037] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A70-0-0] [lt=33] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:50.237335] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:50.237406] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58601][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=17] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:51:50.239002] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:50.241513] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=3855, avg_group_log_size=296) [2025-01-08 00:51:50.243260] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A71-0-0] [lt=75] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:50.252363] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.260931] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61A-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:50.260997] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61A-0-0] [lt=44] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326309849297003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326309849297003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:50.267493] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8801, offset=2500071}, end_cursor:ObLogCursor{file_id=1, log_id=8802, offset=2500376}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c7c050, len:2585, log_data_len:305, seq:8801, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:50.268236] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E71-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:50.268284] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E71-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326309864337001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326309864337001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:50.268875] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58620][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:50.268960] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4855-0-0] [lt=48] init point thread id with(&point=0x5572d50e6840, idx_=1276, point=[thread id=58620, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58620) [2025-01-08 00:51:50.269302] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:50.281444] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CBC-0-0] [lt=51][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:50.283020] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CBC-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:50.285857] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6D0-0-0] [lt=5][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193815199690, cache_obj->added_lc()=false, cache_obj->get_object_id()=13680, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:50.289328] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCB-0-0] [lt=9] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326310400619, keepalive_ts:-1}) [2025-01-08 00:51:50.289362] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCB-0-0] [lt=32] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326310400619, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:50.294654] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:50.294679] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2243195}], scn_array=[cnt:1, {val:1736326309861406004, v:0}], curr_size=122, accum_size=610, time_cost=19163) [2025-01-08 00:51:50.300073] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=100] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8802, offset=2500376}, end_cursor:ObLogCursor{file_id=1, log_id=8803, offset=2500673}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c84050, len:2280, log_data_len:297, seq:8802, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:50.301423] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB61A-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1736326310299944000, start_id=1736326310299944000, end_id=1736326310299944001) [2025-01-08 00:51:50.301553] INFO [LIB] stat (utility.h:1210) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB61A-0-0] [lt=26] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=67, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:50.301930] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] get number(ret=0, service_type_=0, range=1, base_id=1736326310299944000, start_id=1736326310299944001, end_id=1736326310299944002) [2025-01-08 00:51:50.303477] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E71-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:50.303524] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E71-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326309864337001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326309864337001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:50.303986] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=77] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=8434) [2025-01-08 00:51:50.306368] INFO [LIB] stat (utility.h:1210) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B72-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=18, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:50.306399] INFO [LIB] stat (utility.h:1210) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B72-0-0] [lt=30] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4323, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:50.306645] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:50.307160] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A72-0-0] [lt=23] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:50.311965] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6D5-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326310311327000, start_id=1736326310311327000, end_id=1736326310311327001) [2025-01-08 00:51:50.320582] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.320681] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.321059] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=33] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:50.321755] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1210, avg_group_log_size=121) [2025-01-08 00:51:50.321783] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1210) [2025-01-08 00:51:50.324893] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A73-0-0] [lt=104] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:50.327724] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=76] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8803, offset=2500673}, end_cursor:ObLogCursor{file_id=1, log_id=8804, offset=2500970}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c88050, len:1983, log_data_len:297, seq:8803, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:50.329849] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=93] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3787, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:50.329922] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1210) [2025-01-08 00:51:50.329942] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=285, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:50.329943] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8804, offset=2500970}, end_cursor:ObLogCursor{file_id=1, log_id=8805, offset=2501267}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c80050, len:1686, log_data_len:297, seq:8804, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:50.330009] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=42, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:50.338782] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A74-0-0] [lt=48] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:50.340719] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=48] failed to get result(ret=-4008) [2025-01-08 00:51:50.340869] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=69] check new jobs(ret=0, tenant_id=1001, is_oracle_tenant=false, job_infos=[{tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326185605925, start_date_:1736325283345722, last_date_:1736326183353587, this_date_:0, next_date_:1736327083345722, end_date_:64060560000000000, total_:6596354, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736410783345628, last_date_:0, this_date_:0, next_date_:1736410783345628, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:51:50.341121] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=248] check all tenants(ret=0) [2025-01-08 00:51:50.345396] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A76-0-0] [lt=12] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:50.345920] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=48] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-01-08 00:51:50.352829] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.355527] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DA6D-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=36, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:50.355607] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DA6D-0-0] [lt=80] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12192, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:50.355619] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DA6D-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=16, avg_log_submit_to_flush_time=2161, avg_log_submit_to_slide_time=12175) [2025-01-08 00:51:50.387413] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=138] current monitor number(seq_=-1) [2025-01-08 00:51:50.393472] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=50] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:50.398552] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=59] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:50.407506] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:50.409269] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=52] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:50.414903] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=69] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:50.416694] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:50.421580] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1633, avg_group_log_size=163) [2025-01-08 00:51:50.421614] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1633) [2025-01-08 00:51:50.422095] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:50.432152] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD4-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:50.432246] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD4-0-0] [lt=90] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:50.437733] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9933) [2025-01-08 00:51:50.439465] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:50.440096] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=62] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326310099866000, v:0}, new_max_version={val:1736326310099866000, v:0}, last_min_version={val:1736326310029301000, v:0}, last_max_version={val:1736326310029301000, v:0}, total_time_us=7008, delta_us=340226) [2025-01-08 00:51:50.440225] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=124] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326310099866000, v:0}, min_version={val:1736326310099866000, v:0}, max_version={val:1736326310099866000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:50.441582] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E70-0-0] [lt=30] MediumCheckTask(cost_ts=18) [2025-01-08 00:51:50.444533] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] no need to update min/max version(new_version={val:1736326310045734000, v:0}, min_version={val:1736326310045734000, v:0}, max_version={val:1736326310045734000, v:0}) [2025-01-08 00:51:50.452902] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.454911] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1333, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:50.459128] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.459228] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.463115] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=100] gc stale ls task succ [2025-01-08 00:51:50.467820] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4859-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:50.467874] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4859-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:50.467896] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4859-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:50.469791] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1736326310469788, bytes: 882798666, bw: 0.190818 MB/s, add_ts: 1001124, add_bytes: 200312 [2025-01-08 00:51:50.503141] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9933, avg_group_log_size=620) [2025-01-08 00:51:50.503184] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9933) [2025-01-08 00:51:50.504740] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=83] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2660, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:50.504800] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=50] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=127, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:50.504841] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=370] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=56, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:50.507471] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B76-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=270, avg_log_gen_to_submit_time=358, avg_log_submit_to_flush_time=2983, avg_log_submit_to_slide_time=4747) [2025-01-08 00:51:50.507741] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9933, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:50.519339] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.522769] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1633) [2025-01-08 00:51:50.525545] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95D5-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:50.525571] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95D5-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5028, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:50.525808] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1210, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:50.526663] INFO [LIB] stat (utility.h:1210) [58611][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC89A1-0-0] [lt=32] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:50.526697] INFO [LIB] stat (utility.h:1210) [58611][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC89A1-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5295, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:50.526710] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58611][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC89A1-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=61, avg_log_gen_to_submit_time=203, avg_log_submit_to_flush_time=3466, avg_log_submit_to_slide_time=5092) [2025-01-08 00:51:50.527018] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1633, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:50.541776] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:50.541840] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.543961] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.548691] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58608][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D20-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4298) [2025-01-08 00:51:50.548880] INFO [LIB] stat (utility.h:1210) [58608][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D20-0-0] [lt=155] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:50.548954] INFO [LIB] stat (utility.h:1210) [58608][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D20-0-0] [lt=72] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=4443, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:50.548965] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58608][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D20-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=29, avg_log_gen_to_submit_time=134, avg_log_submit_to_flush_time=2005, avg_log_submit_to_slide_time=4308) [2025-01-08 00:51:50.549329] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4176, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:50.550340] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=171] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=1826, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:50.550470] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=124] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=167, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:50.550500] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:50.553070] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.555586] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DA71-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:51:50.598800] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=97] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:50.599201] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C4-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:50.599253] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C4-0-0] [lt=32] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326310361596002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326310361596002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:50.624373] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3034, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:50.624418] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=117, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:50.624530] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=97, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:50.632916] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6D6-0-0] [lt=43][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193815546745, cache_obj->added_lc()=false, cache_obj->get_object_id()=13681, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:50.640116] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:50.644534] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58618][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B78-0-0] [lt=35] init point thread id with(&point=0x5572d50e6880, idx_=1277, point=[thread id=58618, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58618) [2025-01-08 00:51:50.652422] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AFD-0-0] [lt=24] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:50.652494] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3AFD-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:50.653413] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.653927] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1434, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:50.653958] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:50.653982] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=94, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:50.654094] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=67, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:50.663434] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=36] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:50.663505] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=58] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:50.663537] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=27] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=100) [2025-01-08 00:51:50.670785] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BBB-0-0] [lt=42] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:50.670973] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BBD-0-0] [lt=37] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:50.675407] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BED5-0-0] [lt=29] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:50.681326] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4324, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:50.681362] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2243683}], scn_array=[cnt:1, {val:1736326310361596003, v:0}], curr_size=122, accum_size=1098, time_cost=10316) [2025-01-08 00:51:50.681396] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=778, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:50.681516] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=84, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:50.682478] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D2248464D-0-0] [lt=55] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:51:50.706904] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:50.707027] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58597][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=43] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:50.718868] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=53] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:50.733558] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=49] ====== tenant freeze timer task ====== [2025-01-08 00:51:50.742133] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=91, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:50.742281] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=52] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4176) [2025-01-08 00:51:50.744975] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4185) [2025-01-08 00:51:50.749052] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=17] success to update min reserved snapshot(reserved_snapshot=1736324504373137000, duration=1800, snapshot_gc_ts=1736326304373137000) [2025-01-08 00:51:50.753563] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.754067] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=24] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:50.754113] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=42] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:50.754128] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:50.754139] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:50.757556] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=15] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:51:50.758564] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=1003] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:50.759369] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=121] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324504373137000) [2025-01-08 00:51:50.759398] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=26] update reserved snapshot finished(cost_ts=10352, reserved_snapshot=1736324504373137000) [2025-01-08 00:51:50.759499] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=54] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:50.759976] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=466] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:50.759994] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:50.760002] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=8] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:50.760008] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:50.760015] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=95) [2025-01-08 00:51:50.762714] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=41] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:51:50.762741] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=1) [2025-01-08 00:51:50.762752] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:51:50.762762] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:51:50.768270] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A485F-0-0] [lt=20] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=29, avg_log_submit_to_flush_time=5604, avg_log_submit_to_slide_time=57016) [2025-01-08 00:51:50.768452] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:50.768488] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18342, miss_cnt:0, cache_read_size:2243805, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:50.771212] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:50.799093] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=114] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:50.817419] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC419-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:50.818128] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:50.818151] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:50.823020] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:50.823070] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:52986, [delay/us]:prepare:8, schedule:145, submit:104, rt:1569, total:1827) [2025-01-08 00:51:50.823159] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:50.823423] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=179] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52984, [delay/us]: prepare:8, schedule:145, submit:104, device:1569, total:1827) [2025-01-08 00:51:50.823427] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=20] gc stale ls task succ [2025-01-08 00:51:50.823468] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:50.823506] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:50.823552] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:51:50.823522] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4864, iops:16, bw:77441, [delay/us]:prepare:7, schedule:179, submit:363, rt:1866, total:2417) [2025-01-08 00:51:50.823594] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=66] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:50.823697] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=89] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4864, avg_iops:16, avg_bw:77436, [delay/us]: prepare:7, schedule:179, submit:363, device:1866, total:2417) [2025-01-08 00:51:50.823797] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:50.823981] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16299, [delay/us]:prepare:6, schedule:384, submit:1, rt:1758, total:2149) [2025-01-08 00:51:50.824110] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=102] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:37, bw:150771, [delay/us]:prepare:17, schedule:133, submit:468, rt:2070, total:2689) [2025-01-08 00:51:50.824233] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=117] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:50.824412] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=155] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16297, [delay/us]: prepare:6, schedule:384, submit:1, device:1758, total:2149) [2025-01-08 00:51:50.824457] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:37, avg_bw:150745, [delay/us]: prepare:17, schedule:133, submit:468, device:2070, total:2689) [2025-01-08 00:51:50.824478] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:50.824507] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4074, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:816, total:829) [2025-01-08 00:51:50.824535] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:50.824592] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=50] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4074, [delay/us]: prepare:0, schedule:0, submit:600000000, device:816, total:829) [2025-01-08 00:51:50.824613] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:50.824630] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:50.824638] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:50.824645] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:50.824652] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SERVER] net_in=201kB/s, net_out=209kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:50.826480] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0B3-0-0] [lt=74][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:50.829100] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1003}, {id:1}, {id:1002}]) [2025-01-08 00:51:50.831122] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87163-0-0] [lt=30][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:50.831238] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:50.831254] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:50.832097] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:50.840889] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=262] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:50.843501] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58621][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:50.843642] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58621][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B7B-0-0] [lt=30] init point thread id with(&point=0x5572d50e68c0, idx_=1278, point=[thread id=58621, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58621) [2025-01-08 00:51:50.844393] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58622][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:50.844510] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:51:50.845276] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=7] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:50.846918] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=47] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:50.847046] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=50] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326304373137000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:50.847075] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=26] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326304373137000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326304373137000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:50.855407] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:50.857792] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=29] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:50.859510] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=41] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:50.859677] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=89] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326305683208000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:50.859856] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.861487] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B01-0-0] [lt=99][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18332, log_task={header:{begin_lsn:{lsn:2231275}, end_lsn:{lsn:2231397}, log_id:18332, min_scn:{val:1736326310349787004, v:0}, max_scn:{val:1736326310349787004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2231153}, prev_proposal_id:1, committed_end_lsn:{lsn:2231275}, data_checksum:940009244, accum_checksum:2666074066, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326310752149, freeze_ts:1736326310752149, submit_ts:1736326310752155, flushed_ts:1736326310758836, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:6681}, fs_cb_begin_ts=1736326310861473, log_life_time=109324) [2025-01-08 00:51:50.862595] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:50.862670] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-01-08 00:51:50.862684] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:50.862698] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:50.862713] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193814776207, flush_io_finished_ret_:0, flush_io_finished_round_:3797, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:50.865462] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=41] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:50.872835] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4861-0-0] [lt=133] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:50.872886] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4861-0-0] [lt=50] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=61803, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:50.872899] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4861-0-0] [lt=11][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18435, log_task={header:{begin_lsn:{lsn:2243805}, end_lsn:{lsn:2243927}, log_id:18435, min_scn:{val:1736326310361596004, v:0}, max_scn:{val:1736326310361596004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2243683}, prev_proposal_id:1, committed_end_lsn:{lsn:2243805}, data_checksum:1715074492, accum_checksum:2831383626, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326310768198, freeze_ts:1736326310768198, submit_ts:1736326310768213, flushed_ts:1736326310771279, gen_to_freeze cost time:0, gen_to_submit cost time:15, submit_to_flush cost time:3066}, fs_cb_begin_ts=1736326310872821, log_life_time=104623) [2025-01-08 00:51:50.887786] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185449-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:50.888093] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:50.888117] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:51:50.888144] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:50.888160] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:50.888174] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193814799429, flush_io_finished_ret_:0, flush_io_finished_round_:1969, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:50.893635] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:50.895035] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2231575}, scn={val:1736326310848709001, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:50.895041] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=45] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:20, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326310894844, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2231631}, next_to_submit_scn_:{val:1736326310848709002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2231519}, end_lsn:{lsn:2231631}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2231631}, readable_log_tail:{lsn:2231631}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64873137, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326310848709001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326310848709001, v:0}, data_checksum:2627559001, flag:3643}}, init_mode_version:0, accumulate_checksum:729118388, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326310848709001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18244, miss_cnt:0, cache_read_size:2231631, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2231575}, scn_:{val:1736326310848709001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326310905972, first_handle_ts_:1736326310895038, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad8df804150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:51:50.895437] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=44] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324505683208000) [2025-01-08 00:51:50.895479] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=40] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2231575}, scn_:{val:1736326310848709001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326310905972, first_handle_ts_:1736326310895421, replay_cost_:57, retry_cost_:-1, read_log_buf_:0x2ad8df804160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326310894844, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2231631}, next_to_submit_scn_:{val:1736326310848709002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2231519}, end_lsn:{lsn:2231631}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2231631}, readable_log_tail:{lsn:2231631}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64873137, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326310848709001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326310848709001, v:0}, data_checksum:2627559001, flag:3643}}, init_mode_version:0, accumulate_checksum:729118388, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326310848709001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18244, miss_cnt:0, cache_read_size:2231631, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=20, ret=0) [2025-01-08 00:51:50.896614] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.897125] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=86] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.897231] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.897266] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:50.910502] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=77] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:50.911981] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=101] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:50.912074] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326305812845000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:50.912161] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=84] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326305812845000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326305812845000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:50.912363] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=80] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:50.912874] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=48] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:50.915229] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:50.917340] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:51:50.919491] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18728343}, readable_log_tail:{lsn:18728343}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48376425, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:50.919609] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=116] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:50.919627] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2210037}, readable_log_tail:{lsn:2210037}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64894731, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:50.919682] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=54] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:50.919693] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] get_replay_process success(id={id:1002}, submitted_log_size=2244040, unsubmitted_log_size=2244040, replayed_log_size=2244040, unreplayed_log_size=0) [2025-01-08 00:51:50.919702] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] get_replay_process success(id={id:1003}, submitted_log_size=2231631, unsubmitted_log_size=2231631, replayed_log_size=2231631, unreplayed_log_size=0) [2025-01-08 00:51:50.919730] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:50.924812] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:50.953363] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=23] [APPLY STAT CB APPEND COST TIME](cur_stat_count=32, stat_interval=5000000, avg (count/cost)=125, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:50.953418] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=55] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=32, stat_interval=5000000, avg (count/cost)=5202, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:50.953430] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=32, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:50.953497] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=66] [APPLY STAT CB EXECUTE TIME](cur_stat_count=32, stat_interval=5000000, avg (count/cost)=-1016, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:50.953505] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=32, stat_interval=5000000, avg (count/cost)=4311, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:50.959417] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:50.960087] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:50.971657] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEA-0-0] [lt=55] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:50.971745] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEA-0-0] [lt=75] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=91) [2025-01-08 00:51:50.999482] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:51.002887] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=79] get_max_applied_scn(scn={val:1736326310364950004, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2210159}, palf_committed_end_scn_:{val:1736326310865244000, v:0}, last_check_scn_:{val:1736326310865244000, v:0}, max_applied_cb_scn_:{val:1736326310364950004, v:0}}) [2025-01-08 00:51:51.002943] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=53] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2210159}, readable_log_tail:{lsn:2210159}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64894609, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:51.030148] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=42] PNIO [ratelimit] time: 1736326311030143, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007738, add_bytes: 0 [2025-01-08 00:51:51.041810] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=180] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:51.043607] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=61] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1289737, clean_num=31457) [2025-01-08 00:51:51.055140] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B05-0-0] [lt=257][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18335, log_task={header:{begin_lsn:{lsn:2231631}, end_lsn:{lsn:2231753}, log_id:18335, min_scn:{val:1736326310848709002, v:0}, max_scn:{val:1736326310848709002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2231519}, prev_proposal_id:1, committed_end_lsn:{lsn:2231631}, data_checksum:940009244, accum_checksum:3624087029, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326310954426, freeze_ts:1736326310954426, submit_ts:1736326310954447, flushed_ts:1736326310959681, gen_to_freeze cost time:0, gen_to_submit cost time:21, submit_to_flush cost time:5234}, fs_cb_begin_ts=1736326311055125, log_life_time=100699) [2025-01-08 00:51:51.060259] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=167][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:51.080654] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:51.080747] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=85] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:51.080765] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=13] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:51.080853] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=85] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:51.084035] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=27] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:51.086856] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=41] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:51.088537] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=42] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:51.088579] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=41] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:51.090237] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084880-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:51.090322] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084880-0-0] [lt=38] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:51.090340] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084880-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:51.093426] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=14] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:51.097887] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=40] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:51.099540] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=43] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:51.099582] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=41] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:51.101011] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084880-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:51.101103] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084880-0-0] [lt=40] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:51.101121] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084880-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:51.104616] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=13] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:51.106363] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=19] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:51.108758] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBE0B-0-0] [lt=182][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193816022591, cache_obj->added_lc()=false, cache_obj->get_object_id()=13682, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:51.109540] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=86] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:51.111558] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=19] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:51.111600] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=41] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:51.113112] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084880-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:51.113183] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084880-0-0] [lt=18] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:51.113198] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084880-0-0] [lt=13] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:51.113309] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=99] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:51.113897] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=583] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:51.113953] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=46] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:51.113964] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:51.114008] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084880-0-0] [lt=39] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:51.114031] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084880-0-0] [lt=22] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:51.114043] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=7] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:51.114050] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:51.114067] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084880-0-0] [lt=16] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:51.114079] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084880-0-0] [lt=12] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:51.114089] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=6] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:51.114095] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:51.114126] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084880-0-0] [lt=28] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:51.114146] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084880-0-0] [lt=18] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:51.115641] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=10] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:51.115684] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=40] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:51.115696] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=10] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:51.115735] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:51.115748] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=13] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:51.115757] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=9] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:51.115769] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:51.115785] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084880-0-0] [lt=14] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:51.125551] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=75, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:51.125613] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:51.125688] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:51.125710] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:51.125915] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=26] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:288K/69:161120 disk:288K/69:161120 t1m2:40K/9:124794 t1m2g0:40K/9:124794 t1002m2:180K/45:146313 t1002m2g0:180K/45:146313 t1001m2:68K/15:227335 t1001m2g0:68K/15:227335 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:51.128651] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:51.148071] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] no need to update min/max version(new_version={val:1736326310743733000, v:0}, min_version={val:1736326310743733000, v:0}, max_version={val:1736326310743733000, v:0}) [2025-01-08 00:51:51.158023] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=96] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18247, miss_cnt:0, cache_read_size:2231997, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:51.160352] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:51.173935] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=48] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=122, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:51.173984] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=48] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:51.194872] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=24] ====== check clog disk timer task ====== [2025-01-08 00:51:51.194909] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=30] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:51.199787] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=69] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:51.204808] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=33] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:51.228146] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:51.236728] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AE77-0-0] [lt=21] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.817513704299926758e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.517570905390110738e+02) [2025-01-08 00:51:51.242141] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1736326311241071000, start_id=1736326311241071000, end_id=1736326311241071001) [2025-01-08 00:51:51.244032] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:51.247968] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3519, avg_group_log_size=351) [2025-01-08 00:51:51.260834] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=87][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:51.262345] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A4-0-0] [lt=25] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:51.262235] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A82-0-0] [lt=39] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326311262194}) [2025-01-08 00:51:51.262940] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A4-0-0] [lt=34] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326311262913, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326311262913, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326311262913, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:51.263009] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A4-0-0] [lt=68] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326311262194}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326311262913, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326311262913, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326311262913, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:51.271063] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:51.274473] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58582][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=35] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:51.278458] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=27] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:51.281844] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=70] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:51.284407] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:51.284653] WDIAG read_multi_row (ob_table_access_helper.h:238) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:51.286627] INFO delete_row (ob_table_access_helper.h:339) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=30] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326311284683) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326311284683) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:51.286725] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=33] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:51.286834] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=25] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:51:51.286897] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:51:51.288038] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] ====== check clog disk timer task ====== [2025-01-08 00:51:51.288070] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:51.290673] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1C3BBB43-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1736326311287703000, start_id=1736326311287703000, end_id=1736326311287703001) [2025-01-08 00:51:51.300438] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:51.303546] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46498][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6D8-0-0] [lt=33][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193816217379, cache_obj->added_lc()=false, cache_obj->get_object_id()=13683, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:51.310124] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=82, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:51.319720] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=78] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.319780] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.325875] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:51.325900] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:51.330287] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:51.330413] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%", tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-01-08 00:51:51.640701] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=72] no need to update min/max version(new_version={val:1736326311035643000, v:0}, min_version={val:1736326311035643000, v:0}, max_version={val:1736326311035643000, v:0}) [2025-01-08 00:51:51.642051] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1736326311642047, bytes: 882977496, bw: 0.145485 MB/s, add_ts: 1172259, add_bytes: 178830 [2025-01-08 00:51:51.642645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50380][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=0] task scheduled too delay(scheduled_delay=334978, expected_run_ts_us_=1736326311307663, ObClockGenerator::getRealClock()=1736326311642643, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:1, time_interval:100.00ms, expected_run_time:00:51:51.307, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.642758] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:51.643368] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=46] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=486, get_gts_with_stc_cnt=27, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=26, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:51.643407] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=487, get_gts_with_stc_cnt=70, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=69, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:51.643737] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46141][pnio1][T0][YB42C0A89F89-00062B2D18189E44-0-0] [lt=36][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5387, fly_ts=336245, send_timestamp=1736326311307489, peer="192.168.159.139:45058", sz=479) [2025-01-08 00:51:51.643912] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F89-00062B2D16C84A82-0-0] [lt=45][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1472, fly_ts=229005, send_timestamp=1736326311414905, peer="192.168.159.138:42000", sz=540) [2025-01-08 00:51:51.644177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50551][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=78][errcode=0] task scheduled too delay(scheduled_delay=282349, expected_run_ts_us_=1736326311361825, ObClockGenerator::getRealClock()=1736326311644175, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe04190, caller:ob_tenant_freezer.cpp:start:133, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_494, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe04470, time_wheel:0x2ad8dd599990, thread_pool:0x2ad8dd5dd270, is_running:1, time_interval:2.00s, expected_run_time:00:51:51.361, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.644440] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8B-00062B2D16CBB23D-0-0] [lt=18][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1308, fly_ts=323152, send_timestamp=1736326311321286, peer="192.168.159.139:45082", sz=1731) [2025-01-08 00:51:51.644852] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.644896] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:51:51.645127] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=32] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.645537] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] ====== tenant freeze timer task ====== [2025-01-08 00:51:51.645566] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=21] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70495000, tenant_id=1001) [2025-01-08 00:51:51.645587] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:51.646959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=182][errcode=0] task scheduled too delay(scheduled_delay=264432, expected_run_ts_us_=1736326311382524, ObClockGenerator::getRealClock()=1736326311646957, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8a0dff190, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff470, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:1, time_interval:500.00ms, expected_run_time:00:51:51.382, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.647028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=0] task scheduled too delay(scheduled_delay=238496, expected_run_ts_us_=1736326311408530, ObClockGenerator::getRealClock()=1736326311647027, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:51.408, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.647730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=89][errcode=0] task scheduled too delay(scheduled_delay=341264, expected_run_ts_us_=1736326311306463, ObClockGenerator::getRealClock()=1736326311647728, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:51.306, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.647861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50406][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] task scheduled too delay(scheduled_delay=343050, expected_run_ts_us_=1736326311304808, ObClockGenerator::getRealClock()=1736326311647860, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe46190, caller:ob_tenant_freezer.cpp:start:133, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_494, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe46470, time_wheel:0x2ad8cf0e7990, thread_pool:0x2ad8dd4ab4f0, is_running:1, time_interval:2.00s, expected_run_time:00:51:51.304, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.647936] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-01-08 00:51:51.648356] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=57] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6720679936, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:51.649006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] task scheduled too delay(scheduled_delay=123981, expected_run_ts_us_=1736326311525022, ObClockGenerator::getRealClock()=1736326311649004, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:51.525, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.649413] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:51.649549] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] task scheduled too delay(scheduled_delay=300192, expected_run_ts_us_=1736326311349355, ObClockGenerator::getRealClock()=1736326311649548, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:51.349, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.649848] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=56] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:51.649921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=216464, expected_run_ts_us_=1736326311433455, ObClockGenerator::getRealClock()=1736326311649920, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:51.433, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.649982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] task scheduled too delay(scheduled_delay=197682, expected_run_ts_us_=1736326311452299, ObClockGenerator::getRealClock()=1736326311649981, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:51.452, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.649992] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.650018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=160427, expected_run_ts_us_=1736326311489581, ObClockGenerator::getRealClock()=1736326311650009, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff570, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:51.489, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.650044] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.650051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] task scheduled too delay(scheduled_delay=152305, expected_run_ts_us_=1736326311497745, ObClockGenerator::getRealClock()=1736326311650050, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:51.497, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.650070] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.650072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=144971, expected_run_ts_us_=1736326311505100, ObClockGenerator::getRealClock()=1736326311650071, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfc950, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfcc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:51.505, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.650100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=147172, expected_run_ts_us_=1736326311502927, ObClockGenerator::getRealClock()=1736326311650099, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:51.502, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:51.650412] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:51.650439] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-01-08 00:51:51.650450] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:51.650578] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=124] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:51.650594] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193815256558, flush_io_finished_ret_:0, flush_io_finished_round_:1974, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:51.650900] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:51.651404] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:51.651544] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F8B-00062B2D186BE811-0-0] [lt=44][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1315, fly_ts=307447, send_timestamp=1736326311344095, peer="192.168.159.139:45074", sz=239) [2025-01-08 00:51:51.651784] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4015] clock out of order(cur_ts=1736326311651782, cur_ts_=1736326311331064, delta=320718) [2025-01-08 00:51:51.654089] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D170BBDC7-0-0] [lt=21] get number(ret=0, service_type_=0, range=1, base_id=1736326311331064000, start_id=1736326311331064002, end_id=1736326311331064003) [2025-01-08 00:51:51.654189] INFO [LIB] stat (utility.h:1210) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D170BBDC7-0-0] [lt=41] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:51.654584] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=229] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:51:51.654643] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=38] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:51:51.655576] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=31] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=3, inc_cnt=1, queue_size=3) [2025-01-08 00:51:51.657272] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D4F-0-0] [lt=43] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:51.662968] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=118] Slow write(duration=17025, write_len=4096) [2025-01-08 00:51:51.663016] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=45] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8805, offset=2501267}, end_cursor:ObLogCursor{file_id=1, log_id=8806, offset=2501572}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c8c050, len:1389, log_data_len:305, seq:8805, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:51.663120] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [58625][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=0] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:4, threads_idle_time:5298023905, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:51.663427] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD5-0-0] [lt=18] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:51.663478] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD5-0-0] [lt=47] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:51.663522] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=37591, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:51.663777] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=251][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43557451}], scn_array=[cnt:1, {val:1736326311307299000, v:0}], curr_size=589, accum_size=6163, time_cost=353192) [2025-01-08 00:51:51.663818] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=6163) [2025-01-08 00:51:51.663850] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1178, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:51.663913] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=8251) [2025-01-08 00:51:51.663996] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=74] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=42, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:51.664993] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=36998, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:51.665017] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2210525}], scn_array=[cnt:1, {val:1736326311287703000, v:0}], curr_size=122, accum_size=1220, time_cost=339044) [2025-01-08 00:51:51.665035] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:51.665049] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=513, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:51.665219] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=188] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:51.665422] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2244406}], scn_array=[cnt:1, {val:1736326310862856004, v:0}], curr_size=122, accum_size=601, time_cost=388534) [2025-01-08 00:51:51.665497] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=71437, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:51.665514] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84440529}], scn_array=[cnt:1, {val:1736326310946927002, v:0}], curr_size=122, accum_size=1538, time_cost=417417) [2025-01-08 00:51:51.665589] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=538, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:51.665857] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2124) [2025-01-08 00:51:51.666065] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=83, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:51.667991] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [58625][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=98] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5298025550, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:51.668371] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1829F5C6-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1736326311668296000, start_id=1736326311668296000, end_id=1736326311668296001) [2025-01-08 00:51:51.669424] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:51.669381] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58628][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:51.669595] INFO [LIB] stat (utility.h:1210) [58608][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69BF-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:51.669617] INFO [LIB] stat (utility.h:1210) [58608][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69BF-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=64436, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:51.669626] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58608][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69BF-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=502, avg_log_gen_to_submit_time=613, avg_log_submit_to_flush_time=62149, avg_log_submit_to_slide_time=63822) [2025-01-08 00:51:51.669638] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58608][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69BF-0-0] [lt=11][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=45021, log_task={header:{begin_lsn:{lsn:84440529}, end_lsn:{lsn:84440651}, log_id:45021, min_scn:{val:1736326310946927002, v:0}, max_scn:{val:1736326310946927002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84440529}, data_checksum:667309872, accum_checksum:2146707643, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326311247890, freeze_ts:1736326311247898, submit_ts:1736326311247958, flushed_ts:1736326311665845, gen_to_freeze cost time:8, gen_to_submit cost time:68, submit_to_flush cost time:417887}, fs_cb_begin_ts=1736326311669569, log_life_time=421679) [2025-01-08 00:51:51.669775] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:51.669845] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2246, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:51.670238] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:51.670682] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=56050, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:51.670803] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=118][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18728831}], scn_array=[cnt:1, {val:1736326311287703000, v:0}], curr_size=545, accum_size=1399, time_cost=379625) [2025-01-08 00:51:51.670839] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1399) [2025-01-08 00:51:51.670883] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=254, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:51.671364] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=226] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=41, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:51.671585] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95EE-0-0] [lt=152] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:51.671711] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95EE-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1079, avg_group_log_size=119) [2025-01-08 00:51:51.671783] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95EE-0-0] [lt=69] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=967) [2025-01-08 00:51:51.672633] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58628][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D27-0-0] [lt=169] init point thread id with(&point=0x5572d50e6a80, idx_=1285, point=[thread id=58628, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58628) [2025-01-08 00:51:51.673992] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE931-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1736326311673653000, start_id=1736326311673653000, end_id=1736326311673653001) [2025-01-08 00:51:51.674537] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E72-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:51.674613] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E72-0-0] [lt=54] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326310865244002, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326310865244002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:51.675416] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:51:51.680671] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE92F-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1736326311679475000, start_id=1736326311679475000, end_id=1736326311679475001) [2025-01-08 00:51:51.680999] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F5C7-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326311679475000, start_id=1736326311679475000, end_id=1736326311679475001) [2025-01-08 00:51:51.682830] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F89-00062B2D21F89DE1-0-0] [lt=20][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5387, fly_ts=367937, send_timestamp=1736326311314891, peer="192.168.159.139:45062", sz=480) [2025-01-08 00:51:51.684837] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C2BB631-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326311682671000, start_id=1736326311682671000, end_id=1736326311682671001) [2025-01-08 00:51:51.684998] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C2BB631-0-0] [lt=30] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=6120, avg_group_log_size=556) [2025-01-08 00:51:51.685020] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C2BB631-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=6163) [2025-01-08 00:51:51.690827] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CBD-0-0] [lt=52][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:51.691276] INFO [LIB] stat (utility.h:1210) [58621][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EE8-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=22, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:51.691310] INFO [LIB] stat (utility.h:1210) [58621][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EE8-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=34227, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:51.691328] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58621][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EE8-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=102, avg_log_submit_to_flush_time=39743, avg_log_submit_to_slide_time=44922) [2025-01-08 00:51:51.691440] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58621][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EE8-0-0] [lt=108][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=33181, log_task={header:{begin_lsn:{lsn:43557451}, end_lsn:{lsn:43558040}, log_id:33181, min_scn:{val:1736326311307299000, v:0}, max_scn:{val:1736326311307299000, v:0}, data_len:533, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43557451}, data_checksum:3515443691, accum_checksum:1907745289, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326311310043, freeze_ts:1736326311310051, submit_ts:1736326311310151, flushed_ts:1736326311663898, gen_to_freeze cost time:8, gen_to_submit cost time:108, submit_to_flush cost time:353747}, fs_cb_begin_ts=1736326311691168, log_life_time=381125) [2025-01-08 00:51:51.691719] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=101][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326311307299000, v:0}, log_ts:{val:1736326311307299000, v:0}, lsn:{lsn:43557507}, submit_ts:1736326311310035}, this:0x2ad8fb6c4ed8, is_inited_:true, trans_id:{txid:67264}, ls_id:{id:1}, ctx:0x2ad8fb6c14d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=381678, ctx_lock_wait_time=2) [2025-01-08 00:51:51.692233] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5574, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:51.692827] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326310848709003, v:0}, new_max_version={val:1736326310848709003, v:0}, last_min_version={val:1736326310361596003, v:0}, last_max_version={val:1736326310361596003, v:0}, total_time_us=385040, delta_us=844115) [2025-01-08 00:51:51.692936] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=102] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326310848709003, v:0}, min_version={val:1736326310848709003, v:0}, max_version={val:1736326310848709003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:51.694100] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CBD-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:51.696064] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=39] new mysql sessid created(conn.sessid_=3221587001, support_ssl=false) [2025-01-08 00:51:51.696243] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=33] sm conn init succ(conn.sessid_=3221587001, sess.client_addr_="192.168.159.136:39666") [2025-01-08 00:51:51.696448] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=194] accept one succ(*s={this:0x2ad8951f6030, session_id:3221587001, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:183}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:51.696682] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2236093, this={dio_aligned_buf:{buf_write_offset:3895, buf_padding_size:201, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:24483, truncate_used_ts:5528}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2232320, buf_len=122, total_write_size=12172, total_write_size_after_dio=421888, ob_pwrite_used_ts=651848, count=100, each_pwrite_cost=6518) [2025-01-08 00:51:51.696728] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=65217, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:51.696742] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2231997}], scn_array=[cnt:1, {val:1736326310848709005, v:0}], curr_size=122, accum_size=844, time_cost=437203) [2025-01-08 00:51:51.696765] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=844) [2025-01-08 00:51:51.696794] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=92, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:51.696888] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=79, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:51.697749] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=45] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:51.697778] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:292) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=28] mysql login queue(mysql_queue->queue_.size()=1) [2025-01-08 00:51:51.697932] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6A-0-0] [lt=23] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221587001, conn.client_addr_port_=39666, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221587001, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221587001, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D1AF84A6A-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46335, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523361940758729, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:177, fixup_index:-1, fixup_ash_buffer:{this:0x2ad8bc9d7658, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:51:51.700340] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6A-0-0] [lt=73][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:51:51.700412] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6A-0-0] [lt=18] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221587001, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:51.700558] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95F1-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:51.700591] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95F1-0-0] [lt=31] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=55882, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:51.700606] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95F1-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=23, avg_log_gen_to_submit_time=112, avg_log_submit_to_flush_time=49752, avg_log_submit_to_slide_time=55770) [2025-01-08 00:51:51.700621] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95F1-0-0] [lt=14][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=19800, log_task={header:{begin_lsn:{lsn:18728831}, end_lsn:{lsn:18729376}, log_id:19800, min_scn:{val:1736326311287703000, v:0}, max_scn:{val:1736326311287703000, v:0}, data_len:489, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18728831}, data_checksum:3311957715, accum_checksum:4031130964, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326311290856, freeze_ts:1736326311290889, submit_ts:1736326311290931, flushed_ts:1736326311671354, gen_to_freeze cost time:33, gen_to_submit cost time:75, submit_to_flush cost time:380423}, fs_cb_begin_ts=1736326311700544, log_life_time=409688) [2025-01-08 00:51:51.700755] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1399, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:51.700792] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326311287703000, v:0}, log_ts:{val:1736326311287703000, v:0}, lsn:{lsn:18728887}, submit_ts:1736326311290755}, this:0x2ad8c33656d8, is_inited_:true, trans_id:{txid:7551}, ls_id:{id:1}, ctx:0x2ad8c3361cd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=410032, ctx_lock_wait_time=1) [2025-01-08 00:51:51.705944] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58622][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B81-0-0] [lt=23] init point thread id with(&point=0x5572d50e6900, idx_=1279, point=[thread id=58622, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58622) [2025-01-08 00:51:51.707417] INFO [SERVER] process (obmp_init_db.cpp:53) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B95-0-0] [lt=5] init db(db_name=oceanbase) [2025-01-08 00:51:51.708593] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=6][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326311250781000, v:0}, log_ts:{val:1736326311250781000, v:0}, lsn:{lsn:84440707}, submit_ts:1736326311251048}, this:0x2ad89c0a3bd8, is_inited_:true, trans_id:{txid:28190}, ls_id:{id:1}, ctx:0x2ad89c0a01d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=457541, ctx_lock_wait_time=1) [2025-01-08 00:51:51.708950] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=109] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326310946927001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=762015, in_cluster_service=true, cluster_version={val:1736326310865244000, v:0}, min_cluster_version={val:1736326310865244000, v:0}, max_cluster_version={val:1736326310865244000, v:0}, get_cluster_version_err=0, cluster_version_delta=843698, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36816, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=566205929, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:51.718464] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95F0-0-0] [lt=45] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=9, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:51.718490] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95F0-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=54059, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:51.718501] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A95F0-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=18162, log_task={header:{begin_lsn:{lsn:2210525}, end_lsn:{lsn:2210647}, log_id:18162, min_scn:{val:1736326311287703000, v:0}, max_scn:{val:1736326311287703000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2210525}, data_checksum:2222604892, accum_checksum:1785999032, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326311325775, freeze_ts:1736326311325780, submit_ts:1736326311325864, flushed_ts:1736326311665208, gen_to_freeze cost time:5, gen_to_submit cost time:89, submit_to_flush cost time:339344}, fs_cb_begin_ts=1736326311718447, log_life_time=392672) [2025-01-08 00:51:51.718898] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=976, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:51.719982] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=53] [APPLY STAT CB APPEND COST TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=111, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:51.720005] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=24] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=19979, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:51.720013] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:51.720021] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=-1146, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:51.720028] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=18945, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:51.722526] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46493][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6DB-0-0] [lt=23][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193816636351, cache_obj->added_lc()=false, cache_obj->get_object_id()=13684, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:51.725611] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61B-0-0] [lt=247] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:51.725713] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61B-0-0] [lt=69] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326310848709004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326310848709004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:51.728760] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=198] start to print blacklist info [2025-01-08 00:51:51.729630] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=28] ls blacklist refresh finish(cost_time=470185) [2025-01-08 00:51:51.735166] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=20] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8806, offset=2501572}, end_cursor:ObLogCursor{file_id=1, log_id=8807, offset=2501869}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c90050, len:1084, log_data_len:297, seq:8806, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:51.735782] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E72-0-0] [lt=40] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:51.735825] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E72-0-0] [lt=21] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326311287703000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326311287703000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:51.737828] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=49] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:51.737852] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=23] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:51.737974] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84990-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:51.737998] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84990-0-0] [lt=21] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:51.738282] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58629][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:51.743395] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:51.746565] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67] worker thread created(tenant_->id()=1001, group_id_=1, token=8) [2025-01-08 00:51:51.747256] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58630][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:51.747313] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58630][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A486B-0-0] [lt=23] init point thread id with(&point=0x5572d50e6ac0, idx_=1286, point=[thread id=58630, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58630) [2025-01-08 00:51:51.747430] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58630][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A486B-0-0] [lt=33][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18440, log_task={header:{begin_lsn:{lsn:2244406}, end_lsn:{lsn:2244528}, log_id:18440, min_scn:{val:1736326310862856004, v:0}, max_scn:{val:1736326310862856004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2244284}, prev_proposal_id:1, committed_end_lsn:{lsn:2244406}, data_checksum:1715074492, accum_checksum:576845368, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326311276610, freeze_ts:1736326311276610, submit_ts:1736326311276632, flushed_ts:1736326311666621, gen_to_freeze cost time:0, gen_to_submit cost time:22, submit_to_flush cost time:389989}, fs_cb_begin_ts=1736326311747407, log_life_time=470797) [2025-01-08 00:51:51.747724] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=160] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:51.748357] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=134] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2244584}, scn={val:1736326310862856005, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:51:51.748751] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085171-0-0] [lt=35] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:51.749696] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324505683208000) [2025-01-08 00:51:51.749722] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2244584}, scn_:{val:1736326310862856005, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326311772819, first_handle_ts_:1736326311749681, replay_cost_:39, retry_cost_:-1, read_log_buf_:0x2ad95b204240, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326311748391, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2244640}, next_to_submit_scn_:{val:1736326310862856006, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:234, curr_read_buf_start_pos:0, curr_read_buf_end_pos:234, log_storage_:{start_lsn:{lsn:2244406}, end_lsn:{lsn:2244640}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2244640}, readable_log_tail:{lsn:2244640}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64860128, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326310862856005, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326310862856005, v:0}, data_checksum:3336005557, flag:37492}}, init_mode_version:0, accumulate_checksum:3621612831, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326310862856005, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18348, miss_cnt:0, cache_read_size:2244640, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=19, ret=0) [2025-01-08 00:51:51.751685] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=50680, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:51.751716] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=351, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:51.751734] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:51:51.751767] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=84, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:51.751759] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:51.756886] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=99] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8807, offset=2501869}, end_cursor:ObLogCursor{file_id=1, log_id=8808, offset=2502166}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c94050, len:787, log_data_len:297, seq:8807, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:51.758129] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8808, offset=2502166}, end_cursor:ObLogCursor{file_id=1, log_id=8809, offset=2502463}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c98050, len:490, log_data_len:297, seq:8808, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:51.763878] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B0B-0-0] [lt=38] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:51.763940] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B0B-0-0] [lt=50] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=8, avg_log_batch_cnt=0, total_group_log_size=966, avg_group_log_size=120) [2025-01-08 00:51:51.763962] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B0B-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=966) [2025-01-08 00:51:51.764003] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B0B-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:51.764038] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B0B-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=86209, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:51.764060] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B0B-0-0] [lt=21] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=26, avg_log_submit_to_flush_time=46231, avg_log_submit_to_slide_time=86183) [2025-01-08 00:51:51.764077] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B0B-0-0] [lt=14][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18338, log_task={header:{begin_lsn:{lsn:2231997}, end_lsn:{lsn:2232119}, log_id:18338, min_scn:{val:1736326310848709005, v:0}, max_scn:{val:1736326310848709005, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2231875}, prev_proposal_id:1, committed_end_lsn:{lsn:2231997}, data_checksum:940009244, accum_checksum:2057207135, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326311259425, freeze_ts:1736326311259425, submit_ts:1736326311259437, flushed_ts:1736326311696832, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:437395}, fs_cb_begin_ts=1736326311763976, log_life_time=504551) [2025-01-08 00:51:51.764291] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=102] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1088, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:51.767791] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58629][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2EEC-0-0] [lt=25] init point thread id with(&point=0x5572d50e6b00, idx_=1287, point=[thread id=58629, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58629) [2025-01-08 00:51:51.769076] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=223] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:51.770647] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:51.811928] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=87, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:51.811987] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2735) [2025-01-08 00:51:51.817663] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC41C-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:51.817961] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:51.817983] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:51.818243] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3199) [2025-01-08 00:51:51.823382] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=39] ====== check clog disk timer task ====== [2025-01-08 00:51:51.823410] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:51.823547] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:51:51.823708] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:51.823762] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=9] gc stale ls task succ [2025-01-08 00:51:51.824593] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87169-0-0] [lt=10][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:51.824649] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:51.824662] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:51.828363] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:51:51.828481] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%", tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:51:51.829552] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=272] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1003}, {id:1001}, {id:1}]) [2025-01-08 00:51:51.830105] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0B6-0-0] [lt=258][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:51.832391] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=124] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:51.837410] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=27] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:51.837555] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=55] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=183) [2025-01-08 00:51:51.838117] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:51.838173] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=30] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:51.838652] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=473] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=194) [2025-01-08 00:51:51.843826] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=116] get number(ret=0, service_type_=0, range=1, base_id=1736326311840776000, start_id=1736326311840776000, end_id=1736326311840776001) [2025-01-08 00:51:51.844563] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:51.844613] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:51.847854] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A486D-0-0] [lt=38] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=16, avg_log_submit_to_flush_time=51240, avg_log_submit_to_slide_time=95879) [2025-01-08 00:51:51.847916] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A486D-0-0] [lt=60][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18442, log_task={header:{begin_lsn:{lsn:2244640}, end_lsn:{lsn:2244762}, log_id:18442, min_scn:{val:1736326311679475000, v:0}, max_scn:{val:1736326311679475000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2244528}, prev_proposal_id:1, committed_end_lsn:{lsn:2244640}, data_checksum:1715074492, accum_checksum:577363494, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326311747363, freeze_ts:1736326311747363, submit_ts:1736326311747371, flushed_ts:1736326311751736, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:4365}, fs_cb_begin_ts=1736326311847763, log_life_time=100400) [2025-01-08 00:51:51.847938] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=957, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:51.848102] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=151] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18349, miss_cnt:0, cache_read_size:2244762, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:51.850072] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:51.850284] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:51.850500] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:6, bw:23923, [delay/us]:prepare:17, schedule:142, submit:589, rt:70430, total:71178) [2025-01-08 00:51:51.850523] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=1, iops=1, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:51.850583] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:6, avg_bw:23926, [delay/us]: prepare:17, schedule:142, submit:589, device:70430, total:71178) [2025-01-08 00:51:51.850597] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:51.850654] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4840, iops:11, bw:51842, [delay/us]:prepare:12, schedule:31547, submit:298, rt:3093, total:34953) [2025-01-08 00:51:51.850667] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1001, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:51.850753] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4841, avg_iops:11, avg_bw:51845, [delay/us]: prepare:12, schedule:31547, submit:298, device:3093, total:34953) [2025-01-08 00:51:51.850780] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:51.850821] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:15955, [delay/us]:prepare:40, schedule:2412, submit:1, rt:3349, total:5803) [2025-01-08 00:51:51.850854] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4247, iops:26, bw:111690, [delay/us]:prepare:8, schedule:12398, submit:29332, rt:17552, total:59291) [2025-01-08 00:51:51.850969] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=107] [IO STATUS TENANT](tenant_id=1002, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=6, iops=6, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:51.851031] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:15959, [delay/us]: prepare:40, schedule:2412, submit:1, device:3349, total:5803) [2025-01-08 00:51:51.851056] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4248, avg_iops:27, avg_bw:111716, [delay/us]: prepare:8, schedule:12398, submit:29332, device:17552, total:59291) [2025-01-08 00:51:51.851073] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:51.851106] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3989, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:298, total:307) [2025-01-08 00:51:51.851244] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:51.851340] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=86] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3989, [delay/us]: prepare:0, schedule:0, submit:600000000, device:298, total:307) [2025-01-08 00:51:51.851368] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:51.851397] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:51.851452] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:51.851468] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:51.851477] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SERVER] net_in=200kB/s, net_out=192kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:51.851750] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:51.853975] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=957) [2025-01-08 00:51:51.855349] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43561810}], scn_array=[cnt:1, {val:1736326311840776000, v:0}], curr_size=589, accum_size=4359, time_cost=10484) [2025-01-08 00:51:51.856771] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=29] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:51.862926] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:51.863015] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=72] temporary file module use no blocks [2025-01-08 00:51:51.863030] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:51.863044] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:51.863058] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193815776584, flush_io_finished_ret_:0, flush_io_finished_round_:3798, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:51.890297] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918544A-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:51.890603] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=72] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:51.890631] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-01-08 00:51:51.890651] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:51.890664] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:51.890677] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193815802092, flush_io_finished_ret_:0, flush_io_finished_round_:1970, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:51.897237] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.897353] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.897387] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.897657] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:51.914154] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C585498-0-0] [lt=13] InfoPoolResizeTask(cost_ts=14) [2025-01-08 00:51:51.925705] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16E9CA6E-0-0] [lt=86] get number(ret=0, service_type_=0, range=1, base_id=1736326311922542000, start_id=1736326311922542000, end_id=1736326311922542001) [2025-01-08 00:51:51.944722] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:51.956446] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C5-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:51.956853] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C5-0-0] [lt=221] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326311679475001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326311679475001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:52.000143] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get_min_unreplayed_log_info(lsn={lsn:2244884}, scn={val:1736326311679475002, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326311858675, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2244884}, next_to_submit_scn_:{val:1736326311679475002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2244762}, end_lsn:{lsn:2244884}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2245006}, readable_log_tail:{lsn:2245006}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64859762, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326311679475001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326311679475001, v:0}, data_checksum:3348837161, flag:19941}}, init_mode_version:0, accumulate_checksum:2992362809, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326311679475001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18350, miss_cnt:0, cache_read_size:2244884, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:52.034189] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=171] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:52.037647] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=153] PNIO [ratelimit] time: 1736326312037643, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007500, add_bytes: 0 [2025-01-08 00:51:52.045018] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.047303] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4871-0-0] [lt=102] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=23, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:52.047333] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4871-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=85854, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:52.049254] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5299694348, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:52.049372] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:52.049388] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8db9c33e0) [2025-01-08 00:51:52.049398] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5299694348, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:52.051123] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:52.051931] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:52.062518] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:52.063656] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B11-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18341, log_task={header:{begin_lsn:{lsn:2232363}, end_lsn:{lsn:2232485}, log_id:18341, min_scn:{val:1736326311673653002, v:0}, max_scn:{val:1736326311673653002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2232241}, prev_proposal_id:1, committed_end_lsn:{lsn:2232363}, data_checksum:940009244, accum_checksum:510361623, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326311963519, freeze_ts:1736326311963519, submit_ts:1736326311963531, flushed_ts:1736326311967301, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:3770}, fs_cb_begin_ts=1736326312063644, log_life_time=100125) [2025-01-08 00:51:52.082489] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=78][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8951f6030, session_id:3221587001, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326311726020, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1019, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:52.082543] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=45] kill and revert session(conn.sessid_=3221587001, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:51:52.082555] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dfc030, session_id:3221585668, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326300497347, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1693, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:52.082572] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221585668, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:51:52.082580] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8951f6030, session_id:3221587001, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326311726020, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1019, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:52.082634] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221587001, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:51:52.082650] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221587001, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B98-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:11, cur_pkt_pos_:17, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:6741576, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:6741577, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:6741578, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:6741578, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:6741578, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:6741578, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:6741578, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:6741578, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:6741578, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:6741579, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[11]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:6741579, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:6741579, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[13]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:6741579, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[14]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:6741579, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:13}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[15]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:6741579, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:14}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[16]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:6741579, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:15}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:52.082751] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=98] sock ref clean, do destroy(*s={this:0x2ad8a0dfc030, session_id:3221585668, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326300497347, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1693, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:52.082744] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=37] free session successfully(sessid=3221587001) [2025-01-08 00:51:52.082769] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=25] mark session id unused(sessid=3221587001) [2025-01-08 00:51:52.082797] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221585668, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:51:52.082865] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=7] free session successfully(sessid=3221585668) [2025-01-08 00:51:52.082879] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=14] mark session id unused(sessid=3221585668) [2025-01-08 00:51:52.082808] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221585668, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B91-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:25, cur_pkt_pos_:31, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:429795, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:429796, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:429797, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:429797, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:429797, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:429797, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:429797, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:429797, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:429797, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:429798, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[11]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:429798, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:429798, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[13]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:429798, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[14]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:429798, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:13}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[15]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:429798, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:14}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[16]:{send:obp_compess_header_{com_len:662, com_seq:1}, obp20_header_:{payload_len:634, pkt_seq:1, req_id:429798, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:15}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[17]:{receive:obp20_header_:{payload_len:194, pkt_seq:0, req_id:429799, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:190, rec_:194, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[18]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:429799, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[19]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:429799, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[20]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:429799, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[21]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:429799, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[22]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:429799, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[23]:{send:obp_compess_header_{com_len:716, com_seq:1}, obp20_header_:{payload_len:688, pkt_seq:1, req_id:429799, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:13}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[24]:{receive:obp20_header_:{payload_len:410, pkt_seq:0, req_id:429800, flag_.flags:62, ext_flags:0}, obp_mysql_header_{len_:406, rec_:410, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[25]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:429800, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[26]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:429800, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[27]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:429800, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[28]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:429800, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:1, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[29]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:429800, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[30]:{send:obp_compess_header_{com_len:330, com_seq:1}, obp20_header_:{payload_len:302, pkt_seq:1, req_id:429800, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:10}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:52.083252] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=72] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:52.083292] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=95) [2025-01-08 00:51:52.095802] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=190] get_max_applied_scn(scn={val:1736326311992998000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18730287}, palf_committed_end_scn_:{val:1736326311992998000, v:0}, last_check_scn_:{val:1736326311992998000, v:0}, max_applied_cb_scn_:{val:1736326311992998000, v:0}}) [2025-01-08 00:51:52.095910] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=103] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18730287}, readable_log_tail:{lsn:18730287}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48374481, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:52.096074] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=135] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:52.143945] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=72] LogLoopThread round_cost_time(us)(round_cost_time=17) [2025-01-08 00:51:52.144196] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326312143926000, start_id=1736326312143926000, end_id=1736326312143926001) [2025-01-08 00:51:52.144232] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=29] ts response task statistics(alloc_count=60854, free_count=30426) [2025-01-08 00:51:52.144251] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326312144187], gts:1736326312143926000, latest_srr:[mts=1736326312144187]}) [2025-01-08 00:51:52.144269] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:52.144290] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326312144279], gts:1736326312143926000, latest_srr:[mts=1736326312144279]}) [2025-01-08 00:51:52.144539] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326312144527], gts:1736326312143926000, latest_srr:[mts=1736326312144527]}) [2025-01-08 00:51:52.145111] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=257][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.150124] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=71] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:52.150414] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=29] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:212K/48:14450324 disk:212K/48:14450324 t1m2:32K/7:130939 t1m2g0:32K/7:130939 t1002m2:124K/29:11888272 t1002m2g0:124K/29:11888272 t1001m2:56K/12:28929407 t1001m2g0:56K/12:28929407 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:52.150566] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=68] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:51:52.151336] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=108] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:52.153724] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=78] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=45, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:52.153757] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:52.153796] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:52.153816] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:52.154359] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=78] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:52.155557] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=732) [2025-01-08 00:51:52.157685] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58631][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:52.157757] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:51:52.167256] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18253, miss_cnt:0, cache_read_size:2232729, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:52.169761] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:52.169918] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:52.170534] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=61] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:51:52.202316] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:52.214886] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58631][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69C5-0-0] [lt=50] init point thread id with(&point=0x5572d50e6b40, idx_=1288, point=[thread id=58631, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58631) [2025-01-08 00:51:52.244516] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=118] ====== traversal_flush timer task ====== [2025-01-08 00:51:52.244634] INFO [MDS] flush (mds_table_mgr.cpp:195) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58549A-0-0] [lt=30] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326312143926000, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326308847457003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326320087570000, latest_log_ts:{val:1736326303837002001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.244820] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58549A-0-0] [lt=150] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326307358778004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326308847457003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.245005] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58549A-0-0] [lt=154] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326307860790000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326308847457003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.245090] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58549A-0-0] [lt=58] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326307847404000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326308847457003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.245145] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58549A-0-0] [lt=52] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:52.245929] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.252134] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:52.252542] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=208] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:52.256146] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1700) [2025-01-08 00:51:52.257438] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=45, stat_interval=1000000, avg (count/cost)=323, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:52.257771] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=330] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=45, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:52.269368] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58631][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD93-0-0] [lt=43] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326312381952, keepalive_ts:-1}) [2025-01-08 00:51:52.269400] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58631][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD93-0-0] [lt=30] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326312381952, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:52.300628] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=46] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:52.319616] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.319662] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.320866] INFO [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:197) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10] skip repeat study(ret=0, this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326252256096, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, current_timestamp=1736326312320863, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:51:52.322435] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=38] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325772835347000, snapshot_type=1, create_time=1736325772838939, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:52.322461] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325659399768000, snapshot_type=1, create_time=1736325659527637, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:52.322473] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325660825489000, snapshot_type=1, create_time=1736325660889571, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:52.322484] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325772082473004, snapshot_type=2, create_time=1736325772838939, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:52.322494] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325658409784004, snapshot_type=2, create_time=1736325659527637, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:52.322504] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325659910050004, snapshot_type=2, create_time=1736325660889571, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:52.322586] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=75] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325772583377000, snapshot_type=3, create_time=1736325772838939, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:52.322600] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325658978584000, snapshot_type=3, create_time=1736325659527637, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:52.322610] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=1736325660410422000, snapshot_type=3, create_time=1736325660889571, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:52.322620] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325772838939, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:52.322630] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325659527637, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:52.322639] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325660889571, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:52.322870] INFO [MVCC] decide_reserved_snapshot_version_ (ob_multi_version_garbage_collector.cpp:620) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7] succeed to update global reserved snapshot(ret=0, this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326252256096, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, global_reserved_snapshot_={val:1736325658409784004, v:0}, reserved_snapshot={val:1736325658409784004, v:0}) [2025-01-08 00:51:52.322893] INFO [MVCC] refresh_ (ob_multi_version_garbage_collector.cpp:554) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23] multi-version garbage collector refresh successfully(this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326252256096, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, collector={reserved_snapshot_version:{val:1736325658409784004, v:0}, reserved_snapshot_type:2, reserved_status:-1, reserved_create_time:1736325659527637, reserved_addr:"192.168.159.138:2882", is_this_server_disabled:false, status:0}) [2025-01-08 00:51:52.322912] INFO [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:232) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17] repeat refresh successfully(ret=0, this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326312322911, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, current_timestamp=1736326312320904, GARBAGE_COLLECT_RETRY_INTERVAL=60000000) [2025-01-08 00:51:52.322924] INFO [MVCC] repeat_reclaim (ob_multi_version_garbage_collector.cpp:267) [50471][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] skip repeat reclaim(ret=0, this={this:0x2ad8ba621030, last_study_timestamp:1736325772853007, last_refresh_timestamp:1736326312322911, last_reclaim_timestamp:1736325712796862, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325658409784004, v:0}, is_inited:true}, current_timestamp=1736326312322924, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:51:52.346009] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.350395] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=7, avg_log_batch_cnt=1, total_group_log_size=3622, avg_group_log_size=517) [2025-01-08 00:51:52.350521] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=49] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=94, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:52.354435] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=6, avg_log_batch_cnt=1, total_group_log_size=732, avg_group_log_size=122) [2025-01-08 00:51:52.354465] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=732) [2025-01-08 00:51:52.398166] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AF11-0-0] [lt=65] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.817513704299926758e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.517570905390110738e+02) [2025-01-08 00:51:52.429132] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46101][TimerWK3][T1][Y0-0000000000000000-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326312428700000, start_id=1736326312428700000, end_id=1736326312428700001) [2025-01-08 00:51:52.431608] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=24] ====== traversal_flush timer task ====== [2025-01-08 00:51:52.431822] INFO [MDS] flush (mds_table_mgr.cpp:195) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84F09-0-0] [lt=25] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326312244297000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326319284252000, latest_log_ts:{val:1736326302668823001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.431932] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84F09-0-0] [lt=103] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:52.433731] INFO [SHARE] delete_tenant_all (ob_disk_usage_table_operator.cpp:231) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=48] delete successful (ret=0, tenant_id=500) [2025-01-08 00:51:52.433849] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46101][TimerWK3][T1][Y0-0000000000000000-0-0] [lt=43] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:52.435639] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=32] get_stable_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:52.436982] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=37] get_stable_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:52.438831] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [46101][TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-01-08 00:51:52.438936] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=64] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-01-08 00:51:52.438964] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-01-08 00:51:52.439917] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=45] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:52.442385] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B0-0-0] [lt=9][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8baf8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28198} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae19970 trans_version={val:1736322640977792000, v:0} scn={val:1736322640977792000, v:0} tx_id={txid:1449} prev=0x2ad8dae197b8 next=(nil) modify_count=1 acc_checksum=460374033 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312441940000, v:0}, *this={this=0x2ad8dae19658 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad8dae19970 latest_compact_node=(nil) max_trans_version={val:1736326252217943000, v:0} max_trans_id=27791 max_elr_trans_version={val:1736326252217943000, v:0} max_elr_trans_id=27791 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1736322640977792000, v:0} min_modify_scn={val:1736322640977792000, v:0}}) [2025-01-08 00:51:52.442510] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B0-0-0] [lt=122][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8baf8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28198} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342341847, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342341847, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312441940000, v:0}, tx_id:{txid:28198}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28198}, tx_desc:{this:0x2ad8b710fb20, tx_id:{txid:28198}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312441940, active_ts:1736326312441940, commit_ts:-1, finish_ts:-1, timeout_us:29999973, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312546441, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.442607] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B0-0-0] [lt=95][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28198}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.442626] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B0-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28198} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.442715] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B0-0-0] [lt=85][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.442731] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B0-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc540, local_datums:0x2ad9605dc540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad9605dc3f3, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc403, hex: 543811F92D2B0600, int: 1736326312441940}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc40b, hex: 543811F92D2B0600, int: 1736326312441940}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc413, hex: 72D77F0100000000, int: 25155442}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc41b, hex: 00F0120200000000, int: 34795520}]}) [2025-01-08 00:51:52.442838] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:50) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B0-0-0] [lt=59] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=5573, insert_dup_release_count=5573, col_map_alloc_count=0, col_map_release_count=0) [2025-01-08 00:51:52.442870] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B0-0-0] [lt=16][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.442941] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B0-0-0] [lt=8] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28198} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.447088] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=204] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:52.447127] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.452990] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=103] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:52.453281] INFO [COMMON] wash (ob_kvcache_store.cpp:356) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=36] Wash time detail, (compute_wash_size_time=131, refresh_score_time=47, wash_time=0) [2025-01-08 00:51:52.454480] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=46] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1494160, replace_num=15728) [2025-01-08 00:51:52.454509] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:52.456931] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=55] insert successful (ret=0, tenant_id=1, file_type=0, data_size=25155442) [2025-01-08 00:51:52.458829] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.458902] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1335) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B3-0-0] [lt=41][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-01-08 00:51:52.458945] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B3-0-0] [lt=39][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.458962] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B3-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090510, local_datums:0x2ad90f090510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09039c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad90f0903c3, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903d5, hex: 696F11F92D2B0600, int: 1736326312456041}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903dd, hex: 696F11F92D2B0600, int: 1736326312456041}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903e5, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903ed, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:52.459093] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B3-0-0] [lt=62][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.459152] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B3-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28199} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.460149] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.464343] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=9] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:51:52.467065] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=83] insert successful (ret=0, tenant_id=1, file_type=8, data_size=0) [2025-01-08 00:51:52.468858] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B6-0-0] [lt=39][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8bff8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28200} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3d9a8 trans_version={val:1736324383117027001, v:0} scn={val:1736324383117027001, v:0} tx_id={txid:14338} prev=(nil) next=(nil) modify_count=0 acc_checksum=1983279813 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326312466856000, v:0}, *this={this=0x2ad8dae3d918 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3d9a8 latest_compact_node=(nil) max_trans_version={val:1736326252234059000, v:0} max_trans_id=27793 max_elr_trans_version={val:1736326252234059000, v:0} max_elr_trans_id=27793 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383117027001, v:0} min_modify_scn={val:1736324383117027001, v:0}}) [2025-01-08 00:51:52.468930] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B6-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8bff8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28200} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342368512, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342368512, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312466856000, v:0}, tx_id:{txid:28200}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28200}, tx_desc:{this:0x2ad8b7115c40, tx_id:{txid:28200}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312466856, active_ts:1736326312466856, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312582627, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.469040] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B6-0-0] [lt=108][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28200}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.469061] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B6-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28200} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.469091] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B6-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.469102] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B6-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc510, local_datums:0x2ad9605dc510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc39c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3c3, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3d2, hex: A89911F92D2B0600, int: 1736326312466856}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3da, hex: A89911F92D2B0600, int: 1736326312466856}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3e2, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3ea, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:52.469261] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B6-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.469321] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B6-0-0] [lt=16] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28200} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.473195] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=23] insert successful (ret=0, tenant_id=1001, file_type=3, data_size=0) [2025-01-08 00:51:52.474987] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B9-0-0] [lt=88][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8c270 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28201} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3db90 trans_version={val:1736324383129258001, v:0} scn={val:1736324383129258001, v:0} tx_id={txid:14339} prev=(nil) next=(nil) modify_count=0 acc_checksum=450727420 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326312472193000, v:0}, *this={this=0x2ad8dae3db00 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3db90 latest_compact_node=(nil) max_trans_version={val:1736326252238381001, v:0} max_trans_id=27794 max_elr_trans_version={val:1736326252238381001, v:0} max_elr_trans_id=27794 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383129258001, v:0} min_modify_scn={val:1736324383129258001, v:0}}) [2025-01-08 00:51:52.475063] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B9-0-0] [lt=74][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8c270 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28201} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342374692, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342374692, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312472193000, v:0}, tx_id:{txid:28201}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28201}, tx_desc:{this:0x2ad8b7119d00, tx_id:{txid:28201}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312472193, active_ts:1736326312472193, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312586099, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.475148] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B9-0-0] [lt=84][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28201}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.475171] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B9-0-0] [lt=22][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28201} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.475205] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B9-0-0] [lt=32][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.475215] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B9-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090510, local_datums:0x2ad90f090510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09039c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903c3, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903d2, hex: 81AE11F92D2B0600, int: 1736326312472193}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903da, hex: 81AE11F92D2B0600, int: 1736326312472193}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903e2, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903ea, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:52.475689] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B9-0-0] [lt=426][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.475837] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852B9-0-0] [lt=62] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28201} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.479452] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=18] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:52.479486] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:52.479498] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=132, cache_node_num=132) [2025-01-08 00:51:52.479509] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:52.482467] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=132, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}]) [2025-01-08 00:51:52.482813] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=341] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:52.483562] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=29] insert successful (ret=0, tenant_id=1002, file_type=3, data_size=0) [2025-01-08 00:51:52.485306] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BC-0-0] [lt=46][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8c4e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28202} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad96365a690 trans_version={val:1736326252243204000, v:0} scn={val:1736326252243204000, v:0} tx_id={txid:27795} prev=0x2ad96365a4d8 next=(nil) modify_count=121 acc_checksum=3042614252 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312482532000, v:0}, *this={this=0x2ad8dae19d70 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad96365a690 latest_compact_node=0x2ad8dae882a0 max_trans_version={val:1736326252243204000, v:0} max_trans_id=27795 max_elr_trans_version={val:1736326252243204000, v:0} max_elr_trans_id=27795 latest_compact_ts=1736326191874037 last_compact_cnt=119 total_trans_node_cnt=122 max_modify_scn={val:1736326252243204000, v:0} min_modify_scn={val:1736322640988548000, v:0}}) [2025-01-08 00:51:52.485376] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BC-0-0] [lt=68][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8c4e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28202} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342384913, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342384913, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312482532000, v:0}, tx_id:{txid:28202}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28202}, tx_desc:{this:0x2ad8b7117ca0, tx_id:{txid:28202}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312482532, active_ts:1736326312482532, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312594053, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.485491] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BC-0-0] [lt=113][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28202}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.485575] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BC-0-0] [lt=82][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28202} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.485609] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BC-0-0] [lt=32][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.485620] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BC-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc540, local_datums:0x2ad9605dc540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad9605dc3f3, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc403, hex: E4D611F92D2B0600, int: 1736326312482532}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc40b, hex: E4D611F92D2B0600, int: 1736326312482532}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc413, hex: 6FAB080900000000, int: 151563119}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc41b, hex: 6FAB080900000000, int: 151563119}]}) [2025-01-08 00:51:52.485748] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BC-0-0] [lt=79][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.485800] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BC-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5990}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28202} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.490055] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=56] insert successful (ret=0, tenant_id=1, file_type=5, data_size=151563119) [2025-01-08 00:51:52.491891] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BF-0-0] [lt=94][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8c940 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28203} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3e2a0 trans_version={val:1736324383166524001, v:0} scn={val:1736324383166524001, v:0} tx_id={txid:14342} prev=(nil) next=(nil) modify_count=0 acc_checksum=2221000172 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326312489295000, v:0}, *this={this=0x2ad8dae3e210 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3e2a0 latest_compact_node=(nil) max_trans_version={val:1736326252246923001, v:0} max_trans_id=27796 max_elr_trans_version={val:1736326252246923001, v:0} max_elr_trans_id=27796 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383166524001, v:0} min_modify_scn={val:1736324383166524001, v:0}}) [2025-01-08 00:51:52.491959] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BF-0-0] [lt=66][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8c940 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28203} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342391554, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342391554, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312489295000, v:0}, tx_id:{txid:28203}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28203}, tx_desc:{this:0x2ad8b7113be0, tx_id:{txid:28203}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312489295, active_ts:1736326312489295, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312602352, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.492047] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BF-0-0] [lt=86][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28203}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.492088] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BF-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28203} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.492126] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BF-0-0] [lt=36][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.492137] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BF-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090540, local_datums:0x2ad90f090540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903cc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad90f0903f3, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090403, hex: 4FF111F92D2B0600, int: 1736326312489295}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09040b, hex: 4FF111F92D2B0600, int: 1736326312489295}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090413, hex: 710B080000000000, int: 527217}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09041b, hex: 00D03D0000000000, int: 4050944}]}) [2025-01-08 00:51:52.492320] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BF-0-0] [lt=141][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.492499] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852BF-0-0] [lt=13] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28203} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.498890] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=63] insert successful (ret=0, tenant_id=1001, file_type=0, data_size=527217) [2025-01-08 00:51:52.500735] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C2-0-0] [lt=108][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8cbc0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28204} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8da464ac0 trans_version={val:1736324505332247000, v:0} scn={val:1736324505332247000, v:0} tx_id={txid:15233} prev=0x2ad8da464908 next=(nil) modify_count=4 acc_checksum=3222268482 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312497818000, v:0}, *this={this=0x2ad8dae3e400 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8da464ac0 latest_compact_node=(nil) max_trans_version={val:1736326252252997000, v:0} max_trans_id=27797 max_elr_trans_version={val:1736326252252997000, v:0} max_elr_trans_id=27797 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324505332247000, v:0} min_modify_scn={val:1736324383199392001, v:0}}) [2025-01-08 00:51:52.500816] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C2-0-0] [lt=78][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8cbc0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28204} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342400370, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342400370, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312497818000, v:0}, tx_id:{txid:28204}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28204}, tx_desc:{this:0x2ad8b711fe20, tx_id:{txid:28204}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312497818, active_ts:1736326312497818, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312614687, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.500929] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C2-0-0] [lt=111][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28204}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.500957] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C2-0-0] [lt=27][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28204} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.500989] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C2-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.501000] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C2-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc540, local_datums:0x2ad9605dc540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3cc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad9605dc3f3, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc403, hex: 9A1212F92D2B0600, int: 1736326312497818}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc40b, hex: 9A1212F92D2B0600, int: 1736326312497818}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc413, hex: 1ED9080000000000, int: 579870}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc41b, hex: 00802F0000000000, int: 3112960}]}) [2025-01-08 00:51:52.501137] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C2-0-0] [lt=82][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.501258] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C2-0-0] [lt=10] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28204} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.505350] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=51] insert successful (ret=0, tenant_id=1002, file_type=0, data_size=579870) [2025-01-08 00:51:52.507132] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C5-0-0] [lt=40][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8ce48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28205} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3e680 trans_version={val:1736324383216166001, v:0} scn={val:1736324383216166001, v:0} tx_id={txid:14344} prev=(nil) next=(nil) modify_count=0 acc_checksum=3114694747 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326312505254000, v:0}, *this={this=0x2ad8dae3e5f0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3e680 latest_compact_node=(nil) max_trans_version={val:1736326252257312002, v:0} max_trans_id=27799 max_elr_trans_version={val:1736326252257312002, v:0} max_elr_trans_id=27799 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383216166001, v:0} min_modify_scn={val:1736324383216166001, v:0}}) [2025-01-08 00:51:52.507254] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C5-0-0] [lt=120][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8ce48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28205} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342406737, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342406737, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312505254000, v:0}, tx_id:{txid:28205}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28205}, tx_desc:{this:0x2ad8b7101880, tx_id:{txid:28205}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312505254, active_ts:1736326312505254, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312621457, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.507399] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C5-0-0] [lt=143][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28205}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.507420] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C5-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28205} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.507449] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C5-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.507461] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C5-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090510, local_datums:0x2ad90f090510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09039c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad90f0903c3, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903d5, hex: A62F12F92D2B0600, int: 1736326312505254}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903dd, hex: A62F12F92D2B0600, int: 1736326312505254}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903e5, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903ed, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:52.507726] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C5-0-0] [lt=78][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.507865] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C5-0-0] [lt=95] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28205} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.512129] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=30] insert successful (ret=0, tenant_id=1001, file_type=8, data_size=0) [2025-01-08 00:51:52.514101] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C8-0-0] [lt=46][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8d0d8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28206} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3e878 trans_version={val:1736324383240856001, v:0} scn={val:1736324383240856001, v:0} tx_id={txid:14345} prev=(nil) next=(nil) modify_count=0 acc_checksum=3482093884 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326312511307000, v:0}, *this={this=0x2ad8dae3e7e8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3e878 latest_compact_node=(nil) max_trans_version={val:1736326252313883001, v:0} max_trans_id=27800 max_elr_trans_version={val:1736326252313883001, v:0} max_elr_trans_id=27800 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383240856001, v:0} min_modify_scn={val:1736324383240856001, v:0}}) [2025-01-08 00:51:52.514180] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C8-0-0] [lt=77][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8d0d8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28206} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342413808, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342413808, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312511307000, v:0}, tx_id:{txid:28206}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28206}, tx_desc:{this:0x2ad8b710ba60, tx_id:{txid:28206}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312511307, active_ts:1736326312511307, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312628531, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.514363] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C8-0-0] [lt=181][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28206}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.514383] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C8-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28206} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.514412] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C8-0-0] [lt=26][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.514422] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C8-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc510, local_datums:0x2ad9605dc510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc39c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad9605dc3c3, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3d5, hex: 4B4712F92D2B0600, int: 1736326312511307}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3dd, hex: 4B4712F92D2B0600, int: 1736326312511307}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3e5, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3ed, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:52.514514] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C8-0-0] [lt=53][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.514558] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852C8-0-0] [lt=9] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28206} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.519381] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.519924] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=102] insert successful (ret=0, tenant_id=1002, file_type=8, data_size=0) [2025-01-08 00:51:52.524705] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=48] insert successful (ret=0, tenant_id=500, file_type=5, data_size=872415232) [2025-01-08 00:51:52.526352] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852CE-0-0] [lt=78][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8d530 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28208} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad96365b6d8 trans_version={val:1736326252325058003, v:0} scn={val:1736326252325058003, v:0} tx_id={txid:27802} prev=0x2ad96365b520 next=(nil) modify_count=62 acc_checksum=3830524282 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312524856000, v:0}, *this={this=0x2ad8dae3ec78 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad96365b6d8 latest_compact_node=0x2ad8dae88440 max_trans_version={val:1736326252325058003, v:0} max_trans_id=27802 max_elr_trans_version={val:1736326252325058003, v:0} max_elr_trans_id=27802 latest_compact_ts=1736326192026746 last_compact_cnt=59 total_trans_node_cnt=63 max_modify_scn={val:1736326252325058003, v:0} min_modify_scn={val:1736324383282537001, v:0}}) [2025-01-08 00:51:52.526421] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852CE-0-0] [lt=67][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8d530 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28208} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342425954, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342425954, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312524856000, v:0}, tx_id:{txid:28208}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28208}, tx_desc:{this:0x2ad8b7121e80, tx_id:{txid:28208}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312524856, active_ts:1736326312526112, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312641046, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.526530] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852CE-0-0] [lt=106][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28208}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.526550] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852CE-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28208} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.526598] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852CE-0-0] [lt=45][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.526612] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852CE-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc570, local_datums:0x2ad9605dc570}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3fc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc404, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc413, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc41b, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad9605dc423, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc433, hex: 387C12F92D2B0600, int: 1736326312524856}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc43b, hex: 387C12F92D2B0600, int: 1736326312524856}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc443, hex: 9CD7980600000000, int: 110679964}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc44b, hex: 9CD7980600000000, int: 110679964}]}) [2025-01-08 00:51:52.526718] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852CE-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.526792] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852CE-0-0] [lt=30] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28208} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.527086] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58633][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:52.527149] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:51:52.530597] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] insert successful (ret=0, tenant_id=1001, file_type=5, data_size=110679964) [2025-01-08 00:51:52.532015] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D1-0-0] [lt=43][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8d988 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28209} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad96365bb30 trans_version={val:1736326252338010000, v:0} scn={val:1736326252338010000, v:0} tx_id={txid:27803} prev=0x2ad96365b978 next=(nil) modify_count=62 acc_checksum=2149086901 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312531172000, v:0}, *this={this=0x2ad8dae3ee68 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad96365bb30 latest_compact_node=0x2ad8dae884f8 max_trans_version={val:1736326252338010000, v:0} max_trans_id=27803 max_elr_trans_version={val:1736326252338010000, v:0} max_elr_trans_id=27803 latest_compact_ts=1736326192046887 last_compact_cnt=59 total_trans_node_cnt=63 max_modify_scn={val:1736326252338010000, v:0} min_modify_scn={val:1736324383301306001, v:0}}) [2025-01-08 00:51:52.532089] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D1-0-0] [lt=73][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8d988 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28209} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342431721, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342431721, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312531172000, v:0}, tx_id:{txid:28209}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28209}, tx_desc:{this:0x2ad8b71038e0, tx_id:{txid:28209}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312531172, active_ts:1736326312531172, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312643458, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.532179] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D1-0-0] [lt=88][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28209}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.532195] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D1-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28209} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.532278] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D1-0-0] [lt=80][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.532291] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D1-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090570, local_datums:0x2ad90f090570}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903fc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f090404, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090413, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09041b, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad90f090423, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090433, hex: E49412F92D2B0600, int: 1736326312531172}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09043b, hex: E49412F92D2B0600, int: 1736326312531172}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090443, hex: D322841100000000, int: 293872339}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09044b, hex: D322841100000000, int: 293872339}]}) [2025-01-08 00:51:52.532440] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D1-0-0] [lt=107][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.532488] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D1-0-0] [lt=13] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28209} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.536656] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=22] insert successful (ret=0, tenant_id=1002, file_type=5, data_size=293872339) [2025-01-08 00:51:52.538372] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D4-0-0] [lt=48][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8dde0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28210} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae1a6d0 trans_version={val:1736322640997837000, v:0} scn={val:1736322640997837000, v:0} tx_id={txid:1453} prev=0x2ad8dae1a518 next=(nil) modify_count=1 acc_checksum=311300284 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:8} write_epoch=3, snapshot_version={val:1736326312538112000, v:0}, *this={this=0x2ad8dae1a3b8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad8dae1a6d0 latest_compact_node=(nil) max_trans_version={val:1736326252343153001, v:0} max_trans_id=27804 max_elr_trans_version={val:1736326252343153001, v:0} max_elr_trans_id=27804 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1736322640997837000, v:0} min_modify_scn={val:1736322640997837000, v:0}}) [2025-01-08 00:51:52.538456] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D4-0-0] [lt=81][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8dde0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28210} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342438013, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342438013, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312538112000, v:0}, tx_id:{txid:28210}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28210}, tx_desc:{this:0x2ad8b71079a0, tx_id:{txid:28210}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312538112, active_ts:1736326312538112, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312653164, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.538541] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D4-0-0] [lt=83][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28210}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.538610] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D4-0-0] [lt=68][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28210} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.538641] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D4-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.538672] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D4-0-0] [lt=28][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc540, local_datums:0x2ad9605dc540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad9605dc3f3, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc404, hex: 20AB12F92D2B0600, int: 1736326312536864}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc40c, hex: 20AB12F92D2B0600, int: 1736326312536864}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc414, hex: 00C0E20200000000, int: 48414720}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc41c, hex: 00C0E20200000000, int: 48414720}]}) [2025-01-08 00:51:52.538823] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D4-0-0] [lt=113][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.538923] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D4-0-0] [lt=13] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28210} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.541996] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.542416] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27] insert successful (ret=0, tenant_id=1, file_type=7, data_size=48414720) [2025-01-08 00:51:52.543868] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D7-0-0] [lt=51][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8e060 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28211} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae1ab48 trans_version={val:1736322641000286001, v:0} scn={val:1736322641000286001, v:0} tx_id={txid:1454} prev=0x2ad8dae1a990 next=(nil) modify_count=1 acc_checksum=1712141431 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312543250000, v:0}, *this={this=0x2ad8dae1a830 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad8dae1ab48 latest_compact_node=(nil) max_trans_version={val:1736326252354103001, v:0} max_trans_id=27805 max_elr_trans_version={val:1736326252354103001, v:0} max_elr_trans_id=27805 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1736322641000286001, v:0} min_modify_scn={val:1736322641000286001, v:0}}) [2025-01-08 00:51:52.543933] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D7-0-0] [lt=64][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8e060 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28211} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342443509, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342443509, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312543250000, v:0}, tx_id:{txid:28211}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28211}, tx_desc:{this:0x2ad8b711ddc0, tx_id:{txid:28211}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312543250, active_ts:1736326312543250, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312658312, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.544018] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D7-0-0] [lt=83][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28211}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.544034] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D7-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28211} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.544042] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.544060] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D7-0-0] [lt=24][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.544071] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D7-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090540, local_datums:0x2ad90f090540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad90f0903f3, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090403, hex: 12C412F92D2B0600, int: 1736326312543250}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09040b, hex: 12C412F92D2B0600, int: 1736326312543250}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090413, hex: 0060020000000000, int: 155648}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09041b, hex: 0060020000000000, int: 155648}]}) [2025-01-08 00:51:52.544158] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D7-0-0] [lt=51][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.544283] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852D7-0-0] [lt=9] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28211} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.547296] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.547836] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=54] insert successful (ret=0, tenant_id=1, file_type=4, data_size=155648) [2025-01-08 00:51:52.549346] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DA-0-0] [lt=21][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8e2e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28212} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:5} write_epoch=1, prev=this=0x2ad8dae3f5e0 trans_version={val:1736324383421133001, v:0} scn={val:1736324383421133001, v:0} tx_id={txid:14352} prev=(nil) next=(nil) modify_count=0 acc_checksum=2494219223 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326312548356000, v:0}, *this={this=0x2ad8dae3f550 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3f5e0 latest_compact_node=(nil) max_trans_version={val:1736326252364925000, v:0} max_trans_id=27806 max_elr_trans_version={val:1736326252364925000, v:0} max_elr_trans_id=27806 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383421133001, v:0} min_modify_scn={val:1736324383421133001, v:0}}) [2025-01-08 00:51:52.549455] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DA-0-0] [lt=107][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8e2e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28212} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:5} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342449002, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342449002, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312548356000, v:0}, tx_id:{txid:28212}, scn:{branch:0, seq:3}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28212}, tx_desc:{this:0x2ad8b7125f40, tx_id:{txid:28212}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312548356, active_ts:1736326312548356, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312660896, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:5}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.549536] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DA-0-0] [lt=80][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28212}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:5}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.549551] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DA-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28212} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.549577] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DA-0-0] [lt=24][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.549587] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DA-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc540, local_datums:0x2ad9605dc540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3cc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad9605dc3f3, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc404, hex: 04D812F92D2B0600, int: 1736326312548356}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc40c, hex: 04D812F92D2B0600, int: 1736326312548356}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc414, hex: 00D0F70000000000, int: 16240640}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc41c, hex: 00D0F70000000000, int: 16240640}]}) [2025-01-08 00:51:52.549754] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DA-0-0] [lt=131][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.549968] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DA-0-0] [lt=120] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:3759}, to_seq_no={branch:0, seq:4}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28212} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.557138] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=32] insert successful (ret=0, tenant_id=1001, file_type=7, data_size=16240640) [2025-01-08 00:51:52.558899] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DD-0-0] [lt=115][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8e560 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28213} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8da466638 trans_version={val:1736324505384273001, v:0} scn={val:1736324505384273001, v:0} tx_id={txid:15242} prev=0x2ad8da466480 next=(nil) modify_count=4 acc_checksum=590613958 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312557423000, v:0}, *this={this=0x2ad8dae3f740 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8da466638 latest_compact_node=(nil) max_trans_version={val:1736326252372934000, v:0} max_trans_id=27807 max_elr_trans_version={val:1736326252372934000, v:0} max_elr_trans_id=27807 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324505384273001, v:0} min_modify_scn={val:1736324383455288001, v:0}}) [2025-01-08 00:51:52.558970] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DD-0-0] [lt=69][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8e560 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28213} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342458579, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342458579, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312557423000, v:0}, tx_id:{txid:28213}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28213}, tx_desc:{this:0x2ad8b712c060, tx_id:{txid:28213}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312557423, active_ts:1736326312557423, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312670715, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.559060] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DD-0-0] [lt=88][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28213}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.559077] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DD-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28213} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.559106] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DD-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.559128] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DD-0-0] [lt=18][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090540, local_datums:0x2ad90f090540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903cc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad90f0903f3, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090404, hex: 6FFB12F92D2B0600, int: 1736326312557423}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09040c, hex: 6FFB12F92D2B0600, int: 1736326312557423}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090414, hex: 00E0B60000000000, int: 11984896}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09041c, hex: 00E0B60000000000, int: 11984896}]}) [2025-01-08 00:51:52.559300] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DD-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.559359] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852DD-0-0] [lt=14] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28213} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.566026] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=24] insert successful (ret=0, tenant_id=1002, file_type=7, data_size=11984896) [2025-01-08 00:51:52.572614] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:51:52.572739] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58634][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:52.573003] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=42] insert successful (ret=0, tenant_id=500, file_type=4, data_size=45056) [2025-01-08 00:51:52.574738] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E3-0-0] [lt=49][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8e9b0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28215} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae1b190 trans_version={val:1736322641036856001, v:0} scn={val:1736322641036856001, v:0} tx_id={txid:1457} prev=0x2ad8dae1afd8 next=(nil) modify_count=1 acc_checksum=3703409285 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312573627000, v:0}, *this={this=0x2ad8dae1ae78 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad8dae1b190 latest_compact_node=(nil) max_trans_version={val:1736326252381290001, v:0} max_trans_id=27809 max_elr_trans_version={val:1736326252381290001, v:0} max_elr_trans_id=27809 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1736322641036856001, v:0} min_modify_scn={val:1736322641036856001, v:0}}) [2025-01-08 00:51:52.574823] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E3-0-0] [lt=83][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8e9b0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28215} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342474427, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342474427, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312573627000, v:0}, tx_id:{txid:28215}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28215}, tx_desc:{this:0x2ad8b7130120, tx_id:{txid:28215}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312573627, active_ts:1736326312573627, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312687439, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.574999] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E3-0-0] [lt=173][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28215}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.575026] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E3-0-0] [lt=25][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28215} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.575058] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E3-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.575069] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E3-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090540, local_datums:0x2ad90f090540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad90f0903f3, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090403, hex: BB3A13F92D2B0600, int: 1736326312573627}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09040b, hex: BB3A13F92D2B0600, int: 1736326312573627}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090413, hex: 00D0CF0000000000, int: 13619200}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09041b, hex: 00D0CF0000000000, int: 13619200}]}) [2025-01-08 00:51:52.575266] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E3-0-0] [lt=102][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.575355] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E3-0-0] [lt=40] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28215} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.578995] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=54] insert successful (ret=0, tenant_id=1, file_type=1, data_size=13619200) [2025-01-08 00:51:52.580823] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E6-0-0] [lt=41][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8ec30 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28216} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3fed8 trans_version={val:1736324383590655001, v:0} scn={val:1736324383590655001, v:0} tx_id={txid:14358} prev=(nil) next=(nil) modify_count=0 acc_checksum=1106164579 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326312580327000, v:0}, *this={this=0x2ad8dae3fe48 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3fed8 latest_compact_node=(nil) max_trans_version={val:1736326252390022000, v:0} max_trans_id=27810 max_elr_trans_version={val:1736326252390022000, v:0} max_elr_trans_id=27810 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383590655001, v:0} min_modify_scn={val:1736324383590655001, v:0}}) [2025-01-08 00:51:52.580912] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E6-0-0] [lt=88][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8ec30 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28216} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342480475, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342480475, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312580327000, v:0}, tx_id:{txid:28216}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28216}, tx_desc:{this:0x2ad8b7136240, tx_id:{txid:28216}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312580327, active_ts:1736326312580327, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312692050, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.580999] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E6-0-0] [lt=86][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28216}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.581015] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E6-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28216} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.581043] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E6-0-0] [lt=25][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.581052] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E6-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc540, local_datums:0x2ad9605dc540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3cc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad9605dc3f3, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc403, hex: E75413F92D2B0600, int: 1736326312580327}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc40b, hex: E75413F92D2B0600, int: 1736326312580327}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc413, hex: 00E0010000000000, int: 122880}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc41b, hex: 00E0010000000000, int: 122880}]}) [2025-01-08 00:51:52.581176] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E6-0-0] [lt=82][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.581282] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E6-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28216} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.584084] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=55] insert successful (ret=0, tenant_id=1001, file_type=4, data_size=122880) [2025-01-08 00:51:52.585507] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E9-0-0] [lt=39][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8eeb0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28217} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad96365d058 trans_version={val:1736326252395768001, v:0} scn={val:1736326252395768001, v:0} tx_id={txid:27811} prev=0x2ad96365cea0 next=(nil) modify_count=62 acc_checksum=2322107467 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312584346000, v:0}, *this={this=0x2ad8dae40038 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad96365d058 latest_compact_node=0x2ad8dae88680 max_trans_version={val:1736326252395768001, v:0} max_trans_id=27811 max_elr_trans_version={val:1736326252395768001, v:0} max_elr_trans_id=27811 latest_compact_ts=1736326192150383 last_compact_cnt=59 total_trans_node_cnt=63 max_modify_scn={val:1736326252395768001, v:0} min_modify_scn={val:1736324383643182001, v:0}}) [2025-01-08 00:51:52.585571] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E9-0-0] [lt=62][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8eeb0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28217} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342485270, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342485270, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312584346000, v:0}, tx_id:{txid:28217}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28217}, tx_desc:{this:0x2ad8b713e3c0, tx_id:{txid:28217}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312584346, active_ts:1736326312584346, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312697935, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.585686] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E9-0-0] [lt=114][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28217}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.585710] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E9-0-0] [lt=24][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28217} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.585739] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E9-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.585749] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E9-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090540, local_datums:0x2ad90f090540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903cc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad90f0903f3, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090403, hex: 9A6413F92D2B0600, int: 1736326312584346}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09040b, hex: 9A6413F92D2B0600, int: 1736326312584346}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090413, hex: 0030260000000000, int: 2502656}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09041b, hex: 0030260000000000, int: 2502656}]}) [2025-01-08 00:51:52.585832] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E9-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.585866] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852E9-0-0] [lt=9] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28217} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.590799] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=51] insert successful (ret=0, tenant_id=1002, file_type=4, data_size=2502656) [2025-01-08 00:51:52.592419] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852EC-0-0] [lt=85] need check schema(new_schema_version=1736326180221088, cached_tenant_schema_version=1736326180221088, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=INSERT INTO __all_space_usage (tenant_id, svr_ip, svr_port, start_seq, file_type, data_size, used_size) VALUES (?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE data_size = ?, used_size = ?) [2025-01-08 00:51:52.592989] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852EC-0-0] [lt=44] submit log success(service_type_=0, last_id=1736326329254996000, limited_id=1736326329254996000) [2025-01-08 00:51:52.593080] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852EC-0-0] [lt=90] submit log with lock success(service_type_=0, last_id=1736326329254996000, limited_id=1736326329254996000) [2025-01-08 00:51:52.593435] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [46101][TimerWK3][T1][Y0-0000000000000000-0-0] [lt=12] is logging [2025-01-08 00:51:52.593470] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [46101][TimerWK3][T1][Y0-0000000000000000-0-0] [lt=34] submit log with lock(ret=-4023, service_type_=0, last_id=1736326329254996000, limited_id=1736326329254996000) [2025-01-08 00:51:52.597877] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=72] update id meta(ret=0, service_type=0, limited_id=1736326329254996000, latest_scn={val:1736326312592647001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329254996000, latest_log_ts:{val:1736326312592647001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:52.597954] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=76] handle submit callback(service_type_=0, success=true, used_ts=4966, log_ts={val:1736326312592647001, v:0}, limited_id=1736326329254996000) [2025-01-08 00:51:52.597969] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=14] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:52.597978] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:52.609604] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58634][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69DE-0-0] [lt=22] init point thread id with(&point=0x5572d50e6bc0, idx_=1290, point=[thread id=58634, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58634) [2025-01-08 00:51:52.610376] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=11] insert successful (ret=0, tenant_id=500, file_type=1, data_size=0) [2025-01-08 00:51:52.612693] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852ED-0-0] [lt=46][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8f4d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28219} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae40540 trans_version={val:1736324383691496001, v:0} scn={val:1736324383691496001, v:0} tx_id={txid:14361} prev=(nil) next=(nil) modify_count=0 acc_checksum=1965065760 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326312609928000, v:0}, *this={this=0x2ad8dae404b0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae40540 latest_compact_node=(nil) max_trans_version={val:1736326252410054000, v:0} max_trans_id=27813 max_elr_trans_version={val:1736326252410054000, v:0} max_elr_trans_id=27813 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383691496001, v:0} min_modify_scn={val:1736324383691496001, v:0}}) [2025-01-08 00:51:52.612782] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852ED-0-0] [lt=87][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8f4d0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28219} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342511961, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342511961, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312609928000, v:0}, tx_id:{txid:28219}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28219}, tx_desc:{this:0x2ad8b71444e0, tx_id:{txid:28219}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312609928, active_ts:1736326312609928, commit_ts:-1, finish_ts:-1, timeout_us:29999966, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312722708, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.612932] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852ED-0-0] [lt=146][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28219}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.612966] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852ED-0-0] [lt=31][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28219} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.613022] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852ED-0-0] [lt=53][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.613052] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852ED-0-0] [lt=24][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad90f090540, local_datums:0x2ad90f090540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903cc, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad90f0903d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f0903eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad90f0903f3, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090403, hex: 88C813F92D2B0600, int: 1736326312609928}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09040b, hex: 88C813F92D2B0600, int: 1736326312609928}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f090413, hex: 0000BA0000000000, int: 12189696}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad90f09041b, hex: 0000BA0000000000, int: 12189696}]}) [2025-01-08 00:51:52.613240] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852ED-0-0] [lt=76][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.613308] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852ED-0-0] [lt=15] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:4653}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28219} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.619511] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] insert successful (ret=0, tenant_id=1001, file_type=1, data_size=12189696) [2025-01-08 00:51:52.621182] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F0-0-0] [lt=35][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8f750 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28221} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8da467a00 trans_version={val:1736324505450324000, v:0} scn={val:1736324505450324000, v:0} tx_id={txid:15249} prev=0x2ad8da467848 next=(nil) modify_count=4 acc_checksum=3125179850 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326312409057000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326312619399000, v:0}, *this={this=0x2ad8dae406a0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8da467a00 latest_compact_node=(nil) max_trans_version={val:1736326252414747001, v:0} max_trans_id=27814 max_elr_trans_version={val:1736326252414747001, v:0} max_elr_trans_id=27814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324505450324000, v:0} min_modify_scn={val:1736324383710480001, v:0}}) [2025-01-08 00:51:52.621299] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F0-0-0] [lt=115][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8f750 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:28221} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88524e488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326342520858, tablet_id:{id:309}, table_iter:0x2ad88524c978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326342520858, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326312619399000, v:0}, tx_id:{txid:28221}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:28221}, tx_desc:{this:0x2ad8b713c360, tx_id:{txid:28221}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326312619399, active_ts:1736326312619399, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326312734182, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:52.621386] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F0-0-0] [lt=86][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:28221}, data:0x2ad88524b348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:51:52.621403] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F0-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:28221} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:52.621434] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F0-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.621445] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F0-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9605dc540, local_datums:0x2ad9605dc540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3cc, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9605dc3d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc3eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad9605dc3f3, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc403, hex: 87ED13F92D2B0600, int: 1736326312619399}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc40b, hex: 87ED13F92D2B0600, int: 1736326312619399}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc413, hex: 0060870000000000, int: 8871936}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9605dc41b, hex: 0060870000000000, int: 8871936}]}) [2025-01-08 00:51:52.621545] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F0-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.621602] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F0-0-0] [lt=10] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28221} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.629518] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=24] insert successful (ret=0, tenant_id=1002, file_type=1, data_size=8871936) [2025-01-08 00:51:52.631098] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1335) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F3-0-0] [lt=34][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-01-08 00:51:52.631120] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F3-0-0] [lt=19][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:52.631134] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F3-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad9622f64e0, local_datums:0x2ad9622f64e0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad9622f636c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad9622f6374, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad9622f6383, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad9622f638b, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad9622f6393, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad9622f63a2, hex: DA1214F92D2B0600, int: 1736326312628954}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad9622f63aa, hex: DA1214F92D2B0600, int: 1736326312628954}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad9622f63b2, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad9622f63ba, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:52.631332] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F3-0-0] [lt=111][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:52.631390] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46101][TimerWK3][T1][YB42C0A89F89-00062B2D17F852F3-0-0] [lt=14] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:28222} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:52.633382] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB02F-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326312631881000, start_id=1736326312631881000, end_id=1736326312631881001) [2025-01-08 00:51:52.633417] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB02F-0-0] [lt=27] handle gts request statistics(total_rt=1934, total_cnt=279, avg_rt=6.907142857142857473e+00, avg_cnt=2.789999999999999858e+01) [2025-01-08 00:51:52.637268] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=30] insert successful (ret=0, tenant_id=1, file_type=3, data_size=0) [2025-01-08 00:51:52.643546] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1736326312643542, bytes: 883327961, bw: 0.333731 MB/s, add_ts: 1001495, add_bytes: 350465 [2025-01-08 00:51:52.645500] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=48] current monitor number(seq_=-1) [2025-01-08 00:51:52.645775] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46205][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=15] submit log success(service_type_=0, last_id=1736326329284252000, limited_id=1736326329284252000) [2025-01-08 00:51:52.645796] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46205][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=20] submit log with lock success(service_type_=0, last_id=1736326329284252000, limited_id=1736326329284252000) [2025-01-08 00:51:52.645811] INFO [STORAGE.TRANS] process_ (ob_gts_rpc.h:219) [46205][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] gts local cache not updated(result={tenant_id:1001, status:0, srr:[mts=1736326312645668], gts_start:1736326312645069000, gts_end:1736326312645069000}) [2025-01-08 00:51:52.646544] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:51:52.648006] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.650880] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:52.650914] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-01-08 00:51:52.650925] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:52.650952] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:52.650972] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193816564465, flush_io_finished_ret_:0, flush_io_finished_round_:1975, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:52.651257] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:52.652110] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1829F5CC-0-0] [lt=40] is logging [2025-01-08 00:51:52.652131] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1829F5CC-0-0] [lt=19] submit log with lock(ret=-4023, service_type_=0, last_id=1736326329284252000, limited_id=1736326329284252000) [2025-01-08 00:51:52.652513] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:52.653474] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=20] update id meta(ret=0, service_type=0, limited_id=1736326329284252000, latest_scn={val:1736326312645069001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329284252000, latest_log_ts:{val:1736326312645069001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:52.653532] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=57] handle submit callback(service_type_=0, success=true, used_ts=7758, log_ts={val:1736326312645069001, v:0}, limited_id=1736326329284252000) [2025-01-08 00:51:52.653544] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=12] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:52.653554] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:52.653932] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6724874240, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:52.654333] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A487D-0-0] [lt=72][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18450, log_task={header:{begin_lsn:{lsn:2245616}, end_lsn:{lsn:2245738}, log_id:18450, min_scn:{val:1736326312142672004, v:0}, max_scn:{val:1736326312142672004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2245494}, prev_proposal_id:1, committed_end_lsn:{lsn:2245616}, data_checksum:1715074492, accum_checksum:1942651859, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326312550782, freeze_ts:1736326312550782, submit_ts:1736326312550788, flushed_ts:1736326312553657, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:2869}, fs_cb_begin_ts=1736326312654322, log_life_time=103540) [2025-01-08 00:51:52.655781] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:52.655957] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:52.666180] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD6-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:52.666297] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD6-0-0] [lt=113] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:52.666349] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50915][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE937-0-0] [lt=40] get number(ret=0, service_type_=0, range=1, base_id=1736326312665761000, start_id=1736326312665761000, end_id=1736326312665761001) [2025-01-08 00:51:52.670086] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=86] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:52.670600] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:52.671529] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:52.706559] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CBE-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:52.708682] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CBE-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:52.742695] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:52.745748] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58621][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=46] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:51:52.746761] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=35] ====== tenant freeze timer task ====== [2025-01-08 00:51:52.748158] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.753239] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:52.754573] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT APPEND COST TIME](cur_stat_count=35, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:52.756118] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6DD-0-0] [lt=6][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193817669950, cache_obj->added_lc()=false, cache_obj->get_object_id()=13685, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:52.756255] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58630][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=90] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:52.756793] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:52.757017] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=221] try gc retain ctx [2025-01-08 00:51:52.757128] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=109] try to retry start_working [2025-01-08 00:51:52.757271] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=116] finish update min start scn(max_decided_scn={val:1736326312645069000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326302349610005, v:0}, keep_alive_scn_:{val:1736326307358778003, v:0}, update_ts_:1736326312756509}) [2025-01-08 00:51:52.758448] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=147] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326308847457003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326320087570000, latest_log_ts:{val:1736326303837002001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:52.758778] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=135] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1634, avg_group_log_size=163) [2025-01-08 00:51:52.758854] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:51:52.758897] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] finish update min start scn(max_decided_scn={val:1736326312645069000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326302349610004, v:0}, keep_alive_scn_:{val:1736326307358778003, v:0}, update_ts_:1736326312757760}) [2025-01-08 00:51:52.759655] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326307358778004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326308847457003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:52.759828] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=90] finish update min start scn(max_decided_scn={val:1736326312142672004, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326305854841000, v:0}, keep_alive_scn_:{val:1736326310862856000, v:0}, update_ts_:1736326312759604}) [2025-01-08 00:51:52.760527] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326307860790000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326308847457003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:52.760623] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=88] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:52.760649] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:2233339}, scn={val:1736326312162468005, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326312670829, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2233339}, next_to_submit_scn_:{val:1736326312162468005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2233217}, end_lsn:{lsn:2233339}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2233339}, readable_log_tail:{lsn:2233339}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64871429, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326312162468004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326312162468004, v:0}, data_checksum:2124447694, flag:24608}}, init_mode_version:0, accumulate_checksum:921707615, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326312162468004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18258, miss_cnt:0, cache_read_size:2233339, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:52.760746] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=95] finish update min start scn(max_decided_scn={val:1736326312162468004, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326304338519003, v:0}, keep_alive_scn_:{val:1736326309349119001, v:0}, update_ts_:1736326312759604}) [2025-01-08 00:51:52.761526] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326307847404000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326308847457003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:52.764685] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT WRITE LOG COST TIME](cur_stat_count=36, stat_interval=1000000, avg (count/cost)=2509, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:52.764720] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=36, stat_interval=1000000, avg (count/cost)=606, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:52.764767] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=21039) [2025-01-08 00:51:52.764794] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=36, stat_interval=1000000, avg (count/cost)=25, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:52.764879] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=58] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3528, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:52.764897] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1756) [2025-01-08 00:51:52.764915] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=581, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:52.764942] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=23, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:52.765696] INFO [LIB] stat (utility.h:1210) [58634][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69E3-0-0] [lt=235] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=36, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:52.765713] INFO [LIB] stat (utility.h:1210) [58634][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69E3-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=36, stat_interval=1000000, avg (count/cost)=17755, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:52.765723] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58634][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69E3-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=36, avg_log_gen_to_freeze_time=11710, avg_log_gen_to_submit_time=11799, avg_log_submit_to_flush_time=3207, avg_log_submit_to_slide_time=5956) [2025-01-08 00:51:52.766896] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=21039, hit_cnt=36, read_cnt=36, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:52.767464] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2749, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:52.767482] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:52.767497] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=287, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:52.767531] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:52.767542] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6218, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:52.767556] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=6] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=27, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:52.767575] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:52.767695] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8809, offset=2502463}, end_cursor:ObLogCursor{file_id=1, log_id=8810, offset=2502768}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894c9c050, len:4289, log_data_len:305, seq:8809, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:52.771237] INFO [LIB] stat (utility.h:1210) [58585][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC89D9-0-0] [lt=309] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=26, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:52.771313] INFO [LIB] stat (utility.h:1210) [58585][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC89D9-0-0] [lt=76] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6965, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:52.771324] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58585][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC89D9-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=91, avg_log_submit_to_flush_time=4465, avg_log_submit_to_slide_time=6874) [2025-01-08 00:51:52.771411] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1756, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:52.775749] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E73-0-0] [lt=33] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:52.775777] INFO [LIB] stat (utility.h:1210) [58611][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B1F-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:52.775848] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E73-0-0] [lt=35] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326312645069000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326312645069000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:52.775982] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58611][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B1F-0-0] [lt=65] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:52.776248] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58611][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B1F-0-0] [lt=261] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:52.782576] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:491) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E73-0-0] [lt=24] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=0, MAX_GAP=900000000000, new_sync_scn={val:1736326310848709005, v:0}, new_replayable_scn={val:1736326310848709005, v:0}, new_readable_scn={val:1736326310848709005, v:0}, old_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326309849297004, replayable_scn:1736326309849297004, readable_scn:1736326309849297003, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}) [2025-01-08 00:51:52.783053] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E73-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1736326312778269000, start_id=1736326312778269000, end_id=1736326312778269001) [2025-01-08 00:51:52.783366] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E73-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=21, avg_log_batch_cnt=1, total_group_log_size=11568, avg_group_log_size=550) [2025-01-08 00:51:52.783391] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E73-0-0] [lt=20] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10981) [2025-01-08 00:51:52.788459] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2993, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:52.788498] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2233339}], scn_array=[cnt:1, {val:1736326312748474000, v:0}], curr_size=122, accum_size=1342, time_cost=12497) [2025-01-08 00:51:52.788526] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:51:52.788545] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=145, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:52.788638] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:52.789578] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61C-0-0] [lt=57] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:52.789640] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61C-0-0] [lt=35] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326312162468004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326312162468004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:52.792416] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=64] [PALF STAT WRITE LOG COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=3270, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:52.792448] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=12114) [2025-01-08 00:51:52.792469] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=408, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:52.792502] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11570) [2025-01-08 00:51:52.792523] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=28, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:52.794977] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B96-0-0] [lt=37] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=25, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:52.795019] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B96-0-0] [lt=41] [PALF STAT LOG LIFE TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=7661, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:52.795031] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3B96-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=22, avg_log_gen_to_freeze_time=180, avg_log_gen_to_submit_time=288, avg_log_submit_to_flush_time=3806, avg_log_submit_to_slide_time=7372) [2025-01-08 00:51:52.796314] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=12114, hit_cnt=22, read_cnt=22, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:52.802705] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=22] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8810, offset=2502768}, end_cursor:ObLogCursor{file_id=1, log_id=8811, offset=2503065}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ca4050, len:3984, log_data_len:297, seq:8810, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:52.805528] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E73-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:52.805594] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E73-0-0] [lt=42] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326312645069000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326312645069000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:52.816076] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43570154}], scn_array=[cnt:1, {val:1736326312799988000, v:0}], curr_size=589, accum_size=589, time_cost=13430) [2025-01-08 00:51:52.816854] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=138] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326312142672004, v:0}, new_max_version={val:1736326312142672004, v:0}, last_min_version={val:1736326312142672002, v:0}, last_max_version={val:1736326312142672002, v:0}, total_time_us=18501, delta_us=674179) [2025-01-08 00:51:52.817175] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=315] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326312142672004, v:0}, min_version={val:1736326312142672004, v:0}, max_version={val:1736326312142672004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:52.817281] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=35, stat_interval=1000000, avg (count/cost)=73, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:52.817324] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=19964) [2025-01-08 00:51:52.819964] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:52.819988] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:52.824716] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=19964) [2025-01-08 00:51:52.827829] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB61C-0-0] [lt=120] get number(ret=0, service_type_=0, range=1, base_id=1736326312827113000, start_id=1736326312827113000, end_id=1736326312827113001) [2025-01-08 00:51:52.828348] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326302816628, end_time=1736326312828346, dag_status={}) [2025-01-08 00:51:52.828376] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:52.828389] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:52.828398] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:52.828405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:52.828412] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:52.828420] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:52.828427] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:52.828433] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:52.828440] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:52.828448] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828459] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828467] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828475] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828483] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828492] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828499] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828507] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828515] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828523] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828531] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828539] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828548] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828555] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828564] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828572] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828579] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828687] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=105] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828712] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828923] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=164] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828945] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828956] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828964] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828972] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.828980] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829043] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=60] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829057] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829067] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829076] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829084] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829092] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829100] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829108] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829116] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829124] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829132] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829140] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829148] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829156] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829164] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829172] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829263] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=89] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829272] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829280] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829288] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829296] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829304] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829312] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829320] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829328] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829335] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829343] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829351] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829347] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8716F-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:52.829359] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829367] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829375] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829383] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829391] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829399] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829400] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:52.829407] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829412] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:52.829415] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829423] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829431] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829439] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829447] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829455] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829463] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829471] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829479] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829487] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829494] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829503] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829511] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:52.829520] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:52.829528] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:52.829535] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:52.829540] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:52.829546] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:52.829551] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:52.829556] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:52.829561] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:52.829567] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:52.829572] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:52.829577] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:52.829813] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:51:52.830106] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:52.830247] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=9] gc stale ls task succ [2025-01-08 00:51:52.831527] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=40] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}, {id:1003}]) [2025-01-08 00:51:52.832717] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:52.834038] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84649-0-0] [lt=31] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2025-01-08 00:51:52.835532] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84649-0-0] [lt=27] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2025-01-08 00:51:52.835551] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1010) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84649-0-0] [lt=18] no transfer partition job [2025-01-08 00:51:52.835564] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1040) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84649-0-0] [lt=7] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer=true) [2025-01-08 00:51:52.838933] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:284) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84649-0-0] [lt=10] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-01-08 00:51:52.838983] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:493) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84649-0-0] [lt=34] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2025-01-08 00:51:52.839017] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:157) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A84649-0-0] [lt=25] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=3, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1003}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone3", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1736324440080711, last_statistic_bg_stat_time=1736326192603087, last_statistic_schema_version=1736326192037600, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-01-08 00:51:52.839714] INFO pn_ratelimit (group.c:653) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=104] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:52.839903] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=20] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:52.839937] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=17] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=100) [2025-01-08 00:51:52.843665] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC41F-0-0] [lt=30][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:52.849362] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A487F-0-0] [lt=36] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:52.849491] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A487F-0-0] [lt=102] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:51:52.849571] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A487F-0-0] [lt=76] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1454) [2025-01-08 00:51:52.849649] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A487F-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=26, avg_log_submit_to_flush_time=3028, avg_log_submit_to_slide_time=48560) [2025-01-08 00:51:52.849667] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A487F-0-0] [lt=16][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18451, log_task={header:{begin_lsn:{lsn:2245738}, end_lsn:{lsn:2245860}, log_id:18451, min_scn:{val:1736326312645069001, v:0}, max_scn:{val:1736326312645069001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2245616}, prev_proposal_id:1, committed_end_lsn:{lsn:2245738}, data_checksum:1715074492, accum_checksum:890109134, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326312654291, freeze_ts:1736326312654291, submit_ts:1736326312654299, flushed_ts:1736326312658224, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:3925}, fs_cb_begin_ts=1736326312849633, log_life_time=195342) [2025-01-08 00:51:52.850073] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:52.850041] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=649809920, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=423931904, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:52.850108] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18358, miss_cnt:0, cache_read_size:2245860, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:52.850145] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=64] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:52.850321] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=171] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2886, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:52.850433] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1116487680, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2104737792, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:52.850546] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=64] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:52.850573] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:52.850912] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=264][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.851824] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853BF-0-0] [lt=27] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:52.851871] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853BF-0-0] [lt=39] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:52.852032] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=72][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43570743}], scn_array=[cnt:1, {val:1736326312827113000, v:0}], curr_size=750, accum_size=1339, time_cost=20603) [2025-01-08 00:51:52.852525] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:52.853944] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2353, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:52.853972] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:52.853992] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=777, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:52.854099] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=77, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:52.854853] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:52.857741] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=28] success to update min reserved snapshot(reserved_snapshot=1736324505812845000, duration=1800, snapshot_gc_ts=1736326305812845000) [2025-01-08 00:51:52.862098] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0B9-0-0] [lt=23][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:52.863512] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:52.863565] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:35, bw:161705, [delay/us]:prepare:10, schedule:67, submit:267, rt:2015, total:2360) [2025-01-08 00:51:52.863581] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:52.863635] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:35, avg_bw:161728, [delay/us]: prepare:10, schedule:67, submit:267, device:2015, total:2360) [2025-01-08 00:51:52.863649] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:52.863706] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:20, bw:88951, [delay/us]:prepare:15, schedule:646, submit:300, rt:2468, total:3430) [2025-01-08 00:51:52.863720] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:52.863758] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:20, avg_bw:88954, [delay/us]: prepare:15, schedule:646, submit:300, device:2468, total:3430) [2025-01-08 00:51:52.863770] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:52.863794] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:6144, iops:2, bw:12130, [delay/us]:prepare:29, schedule:163, submit:1, rt:4434, total:4628) [2025-01-08 00:51:52.863816] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4403, iops:39, bw:173871, [delay/us]:prepare:8, schedule:229, submit:115, rt:2405, total:2759) [2025-01-08 00:51:52.863825] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:52.863853] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:6144, avg_iops:2, avg_bw:12132, [delay/us]: prepare:29, schedule:163, submit:1, device:4434, total:4628) [2025-01-08 00:51:52.863869] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4403, avg_iops:40, avg_bw:173899, [delay/us]: prepare:8, schedule:229, submit:115, device:2405, total:2759) [2025-01-08 00:51:52.863879] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:52.863898] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4044, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:871, total:879) [2025-01-08 00:51:52.863922] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:52.863948] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4044, [delay/us]: prepare:0, schedule:0, submit:600000000, device:871, total:879) [2025-01-08 00:51:52.863966] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:52.863982] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:52.863990] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:52.863998] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:52.864006] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=353kB/s, net_out=306kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:52.869984] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=48] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:52.870101] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8811, offset=2503065}, end_cursor:ObLogCursor{file_id=1, log_id=8812, offset=2503362}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ca0050, len:3687, log_data_len:297, seq:8811, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:52.870960] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:52.871052] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=516] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:52.872095] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=138] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8812, offset=2503362}, end_cursor:ObLogCursor{file_id=1, log_id=8813, offset=2503659}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ca8050, len:3390, log_data_len:297, seq:8812, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:52.872712] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB APPEND COST TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=164, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:52.872734] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=21] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=4990, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:52.872741] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:52.872748] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB EXECUTE TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=-814, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:52.872774] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=25] [APPLY STAT CB TOTAL TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=4340, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:52.881509] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=31] update reserved snapshot finished(cost_ts=23771, reserved_snapshot=1736324505812845000) [2025-01-08 00:51:52.881866] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=37] ====== traversal_flush timer task ====== [2025-01-08 00:51:52.882040] INFO [MDS] flush (mds_table_mgr.cpp:195) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D1738571B-0-0] [lt=57] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326312813632000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329254996000, latest_log_ts:{val:1736326312592647001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.882173] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D1738571B-0-0] [lt=127] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:52.892621] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918544B-0-0] [lt=53][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:52.893125] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=49] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:52.893156] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-01-08 00:51:52.893167] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:52.893181] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:52.893243] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=60] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193816804572, flush_io_finished_ret_:0, flush_io_finished_round_:1971, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:52.896439] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.897058] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.897097] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.897121] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:52.923153] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46379][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857CA-0-0] [lt=28][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193817836982, cache_obj->added_lc()=false, cache_obj->get_object_id()=13686, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:52.924412] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46386][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857CB-0-0] [lt=31][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:52.926091] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857CD-0-0] [lt=39][errcode=-4201] hashtable not init [2025-01-08 00:51:52.951078] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:52.958605] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=42] ====== checkpoint timer task ====== [2025-01-08 00:51:52.960014] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=30] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:52.960059] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=45] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:52.960130] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326309849297004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326320087570000, latest_log_ts:{val:1736326303837002001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.960210] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=77] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:52.960287] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:52.960308] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] get rec log scn(service_type_=0, rec_log_ts={val:1736326303837002001, v:0}) [2025-01-08 00:51:52.960316] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:52.960331] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:52.960350] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:52.960363] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:52.960483] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:52.960561] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=76] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:52.961295] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:52.961332] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=37] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:52.961376] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326307358778004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326309849297004, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.961441] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=63] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:52.961455] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:52.961470] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:52.961485] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:52.961501] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2211980}, scn={val:1736326312645069002, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:52.963767] INFO [PALF] pread (log_reader.cpp:130) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1524) [2025-01-08 00:51:52.964152] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=70] BlockGCTimerTask success(ret=0, cost_time_us=16, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:52.964810] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B21-0-0] [lt=30] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=30, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:52.964837] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B21-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29401, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:52.964849] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B21-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=3182, avg_log_submit_to_slide_time=29379) [2025-01-08 00:51:52.964953] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B21-0-0] [lt=102][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18349, log_task={header:{begin_lsn:{lsn:2233339}, end_lsn:{lsn:2233461}, log_id:18349, min_scn:{val:1736326312748474000, v:0}, max_scn:{val:1736326312748474000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2233217}, prev_proposal_id:1, committed_end_lsn:{lsn:2233339}, data_checksum:940009244, accum_checksum:2269864483, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326312775763, freeze_ts:1736326312775764, submit_ts:1736326312775844, flushed_ts:1736326312788574, gen_to_freeze cost time:1, gen_to_submit cost time:81, submit_to_flush cost time:12730}, fs_cb_begin_ts=1736326312964762, log_life_time=188999) [2025-01-08 00:51:52.964966] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:52.965828] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8813, offset=2503659}, end_cursor:ObLogCursor{file_id=1, log_id=8814, offset=2503956}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cac050, len:3093, log_data_len:297, seq:8813, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:52.968829] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF34-0-0] [lt=27] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326312983900, keepalive_ts:-1}) [2025-01-08 00:51:52.969059] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF34-0-0] [lt=229] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326312983900, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:52.969037] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=159] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326312645069002, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:52.969133] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:52.969157] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:52.970086] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:52.970124] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=39] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:52.970184] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326307860790000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326310848709005, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.970408] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=220] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:52.970428] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:52.970448] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:52.970465] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:52.970481] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2245982}, scn={val:1736326312811284000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:52.974675] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEB-0-0] [lt=70] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:52.974757] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEB-0-0] [lt=72] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=85) [2025-01-08 00:51:52.975451] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58635][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:52.975448] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:52.975793] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=40] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8814, offset=2503956}, end_cursor:ObLogCursor{file_id=1, log_id=8815, offset=2504253}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cb4050, len:2796, log_data_len:297, seq:8814, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:52.976018] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326312811284000, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:52.976062] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:52.976088] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:52.977636] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:52.977686] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=50] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:52.977795] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=74] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326307847404000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326310848709005, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:52.977981] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=149] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:52.978008] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=51] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:52.978031] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:52.978049] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:52.978130] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=71] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2233583}, scn={val:1736326312813632000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:52.980431] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=58] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8815, offset=2504253}, end_cursor:ObLogCursor{file_id=1, log_id=8816, offset=2504550}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cb0050, len:2499, log_data_len:297, seq:8815, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:52.980514] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326312813632000, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:52.980553] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:52.980578] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:52.980590] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:52.988167] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C6-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:52.988288] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C6-0-0] [lt=95] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326312811284000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326312811284000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:53.008823] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=53] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:51:53.008885] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=59] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=1) [2025-01-08 00:51:53.008900] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:51:53.008910] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=7] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:51:53.045434] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1736326313045427, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007784, add_bytes: 0 [2025-01-08 00:51:53.045582] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=43] runTimer to refresh locality_info(ret=0) [2025-01-08 00:51:53.047994] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:346) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=20] set_server_locality_array success(server_locality_array=[{inited:true, addr:"192.168.159.137:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322580423131, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.138:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581270010, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.139:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581175973, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-01-08 00:51:53.048055] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=48] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-01-08 00:51:53.048118] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=61] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-01-08 00:51:53.048137] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=10] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1001, primary_zone=RANDOM) [2025-01-08 00:51:53.048144] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=7] add_locality_zone success(locality_zone={tenant_id:1001, region_priority:18446744073709551615}) [2025-01-08 00:51:53.048158] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1002, primary_zone=RANDOM) [2025-01-08 00:51:53.048166] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=7] add_locality_zone success(locality_zone={tenant_id:1002, region_priority:18446744073709551615}) [2025-01-08 00:51:53.048176] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1001, region_priority:18446744073709551615}, {tenant_id:1002, region_priority:18446744073709551615}]}) [2025-01-08 00:51:53.050625] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5303507344, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:53.050801] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=68] recycle one thread(this=0x2ad8885abcf0, total=5, remain=4) [2025-01-08 00:51:53.050818] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5303507344, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:53.052522] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=153][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.056311] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6731165696, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:53.056371] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185138-0-0] [lt=11] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326308387862, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:53.056421] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185138-0-0] [lt=48] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326308390361, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:53.056477] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185138-0-0] [lt=53] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, renew_time:1736326308391848, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, renew_time:1736326308391849, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, renew_time:1736326308391851, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326308391844, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:51:53.062301] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1001, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18174, max_lsn:2212102, max_scn:{val:1736326312645069003, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2212102}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2211980}, last_submit_end_lsn:{lsn:2212102}, last_submit_log_id:18174, last_submit_log_pid:1, max_flushed_lsn:{lsn:2211980}, max_flushed_end_lsn:{lsn:2212102}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2212102}, last_slide_log_id:18174, last_slide_scn:{val:1736326312645069003, v:0}, last_slide_lsn:{lsn:2211980}, last_slide_end_lsn:{lsn:2212102}, last_slide_log_pid:1, last_slide_log_accum_checksum:3444958869, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924f6a530}, [StateMgr]={this:0x2ad924f6e830, self:"192.168.159.137:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18175, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324447115984, palf_role_change_cb:0x2ad924f752f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924f6bc70}, [ModeMgr]={palf_id:1001, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324447762546, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436756804001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/meta", dir_fd:177, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924f70a38, flashback_version:0}, log_storage:{log_tail:{lsn:2212102}, readable_log_tail:{lsn:2212102}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64892666, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1736324447114985, this:0x2ad924f6ef70}, [Reconfirm]={palf_id:1001, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:1"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324447117136, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924f6e9b0}) [2025-01-08 00:51:53.062663] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=356] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326312968074, last_flushed_end_lsn:{lsn:2212102}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326312968674, last_flushed_end_lsn:{lsn:2212102}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326312972383, last_flushed_end_lsn:{lsn:2212102}}]) [2025-01-08 00:51:53.062689] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1001, self="192.168.159.137:2882", log_id=18175, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326312968074, last_flushed_end_lsn:{lsn:2212102}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326312968674, last_flushed_end_lsn:{lsn:2212102}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326312972383, last_flushed_end_lsn:{lsn:2212102}}]) [2025-01-08 00:51:53.062863] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BBB-0-0] [lt=120] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:53.063114] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BBD-0-0] [lt=19] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:53.071233] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=112] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:53.083882] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58585][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D0-0-0] [lt=11] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326313101067, keepalive_ts:-1}) [2025-01-08 00:51:53.083990] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58585][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D0-0-0] [lt=105] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326313101067, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:53.085146] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D1-0-0] [lt=28] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326313101214, keepalive_ts:-1}) [2025-01-08 00:51:53.085424] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D1-0-0] [lt=276] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326313101214, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:53.114397] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:801) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F852F6-0-0] [lt=21] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, location:{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, location:{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}]) [2025-01-08 00:51:53.146553] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:53.148558] INFO [LIB] stat (utility.h:1210) [58615][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4883-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=43, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:53.148592] INFO [LIB] stat (utility.h:1210) [58615][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4883-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=49930, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:53.151519] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:53.152841] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.153621] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:53.156272] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:53.157056] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:448K/102:312904 disk:448K/102:312904 t1m2:160K/36:60002 t1m2g0:160K/36:60002 t1002m2:200K/47:322648 t1002m2g0:200K/47:322648 t1001m2:88K/19:767408 t1001m2g0:88K/19:767408 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:53.161613] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:53.162445] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326309121643, last_first_register_time_us=1736324449183670, register_time_us=1736326309121643, curr_time_us=1736326313162442) [2025-01-08 00:51:53.162512] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=35] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326313162481, last_submit_register_req_time_us=1736326313162481, register_parent_reason=1) [2025-01-08 00:51:53.162529] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:53.162952] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=69, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:53.162982] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:53.163029] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:53.163047] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:53.164605] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=338] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:53.170612] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:53.170851] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=301] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:53.171884] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:53.182608] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:53.186878] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3782) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-01-08 00:51:53.186893] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3805) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-01-08 00:51:53.195217] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] ====== check clog disk timer task ====== [2025-01-08 00:51:53.195288] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=63] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:53.235015] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852B8-0-0] [lt=36] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:53.249363] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF6C-0-0] [lt=15] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326313364729, keepalive_ts:-1}) [2025-01-08 00:51:53.249406] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF6C-0-0] [lt=43] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326313364729, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:53.249744] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1736326313005452000, v:0}, base_version={val:1736326312799988000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326313044926000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326313249722, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326313005452000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326313269146, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326313044926000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326313184612, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:53.253083] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=185][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.257113] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=140] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:53.259010] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=205] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1321194, clean_num=31457) [2025-01-08 00:51:53.264889] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:53.268271] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B27-0-0] [lt=154][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18352, log_task={header:{begin_lsn:{lsn:2233705}, end_lsn:{lsn:2233827}, log_id:18352, min_scn:{val:1736326312813632002, v:0}, max_scn:{val:1736326312813632002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2233583}, prev_proposal_id:1, committed_end_lsn:{lsn:2233705}, data_checksum:940009244, accum_checksum:3462599883, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326313166172, freeze_ts:1736326313166172, submit_ts:1736326313166242, flushed_ts:1736326313168372, gen_to_freeze cost time:0, gen_to_submit cost time:70, submit_to_flush cost time:2130}, fs_cb_begin_ts=1736326313268259, log_life_time=102087) [2025-01-08 00:51:53.268323] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=56] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18262, miss_cnt:0, cache_read_size:2233827, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:53.268624] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=186] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=84, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:53.268649] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:53.269275] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58631][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FD-0-0] [lt=15] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326313286739, keepalive_ts:-1}) [2025-01-08 00:51:53.269382] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58631][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FD-0-0] [lt=95] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326313286739, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:53.271702] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=131] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:53.288671] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] ====== check clog disk timer task ====== [2025-01-08 00:51:53.288712] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=33] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:53.300791] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:53.319667] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.319741] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.353417] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.356938] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=221] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=36, avg_log_batch_cnt=1, total_group_log_size=19653, avg_group_log_size=545) [2025-01-08 00:51:53.361999] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=92, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:53.363449] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:51:53.363497] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:51:53.400800] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AF20-0-0] [lt=15] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.817513704299926758e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.517570905390110738e+02) [2025-01-08 00:51:53.404303] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=53] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:53.453713] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.458924] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.459353] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.460611] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=58] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:53.463492] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:53.463528] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=28] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70914440, tenant_id=1001) [2025-01-08 00:51:53.463553] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:53.464014] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] ====== tenant freeze timer task ====== [2025-01-08 00:51:53.465101] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:53.465615] WDIAG read_multi_row (ob_table_access_helper.h:238) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=188][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:53.468025] INFO delete_row (ob_table_access_helper.h:339) [50585][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=19] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1001, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326313465635) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326313465635) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:53.472091] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:53.519053] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.528988] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] clear dtl interm result cost(us)(clear_cost=7102, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:53.541974] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.543688] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.555845] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.561181] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:53.564416] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58634][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-01-08 00:51:53.644610] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1736326313644605, bytes: 883606180, bw: 0.265049 MB/s, add_ts: 1001063, add_bytes: 278219 [2025-01-08 00:51:53.644776] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F5D3-0-0] [lt=52] submit log success(service_type_=0, last_id=1736326330087570000, limited_id=1736326330087570000) [2025-01-08 00:51:53.644798] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F5D3-0-0] [lt=21] submit log with lock success(service_type_=0, last_id=1736326330087570000, limited_id=1736326330087570000) [2025-01-08 00:51:53.646016] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=92] current monitor number(seq_=-1) [2025-01-08 00:51:53.646595] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=36] gc stale ls task succ [2025-01-08 00:51:53.647746] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:109) [46205][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=69] is logging [2025-01-08 00:51:53.647772] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:57) [46205][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=24] submit log with lock(ret=-4023, service_type_=0, last_id=1736326330087570000, limited_id=1736326330087570000) [2025-01-08 00:51:53.648250] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=10] update id meta(ret=0, service_type=0, limited_id=1736326330087570000, latest_scn={val:1736326313644270001, v:0}, *this={tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326309849297004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326330087570000, latest_log_ts:{val:1736326313644270001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:51:53.648373] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=121] handle submit callback(service_type_=0, success=true, used_ts=3598, log_ts={val:1736326313644270001, v:0}, limited_id=1736326330087570000) [2025-01-08 00:51:53.648387] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=13] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:51:53.648396] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:51:53.651902] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:53.654301] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:53.656382] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.657255] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:53.663776] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BBF-0-0] [lt=14] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:53.664059] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D50-0-0] [lt=22] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:53.666307] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:53.667828] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD7-0-0] [lt=14] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:53.667878] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD7-0-0] [lt=46] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:53.671175] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=50] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:53.671329] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:53.672435] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=48] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:51:53.672888] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=520] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:53.673028] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=26][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:53.673082] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=12][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:53.673707] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B2F-0-0] [lt=33][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18356, log_task={header:{begin_lsn:{lsn:2234193}, end_lsn:{lsn:2234315}, log_id:18356, min_scn:{val:1736326313163837003, v:0}, max_scn:{val:1736326313163837003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2234071}, prev_proposal_id:1, committed_end_lsn:{lsn:2234193}, data_checksum:940009244, accum_checksum:2034888619, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326313573672, freeze_ts:1736326313573672, submit_ts:1736326313573686, flushed_ts:1736326313576061, gen_to_freeze cost time:0, gen_to_submit cost time:14, submit_to_flush cost time:2375}, fs_cb_begin_ts=1736326313673692, log_life_time=100020) [2025-01-08 00:51:53.674018] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=86][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:53.677603] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58636][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:53.677665] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58636][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F0E-0-0] [lt=30] init point thread id with(&point=0x5572d50e6c40, idx_=1292, point=[thread id=58636, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58636) [2025-01-08 00:51:53.680529] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] worker thread created(tenant_->id()=1001, group_id_=1, token=8) [2025-01-08 00:51:53.681254] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58635][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A962B-0-0] [lt=39] init point thread id with(&point=0x5572d50e6c00, idx_=1291, point=[thread id=58635, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58635) [2025-01-08 00:51:53.692658] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BED6-0-0] [lt=93] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:53.699901] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:194) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=34] get wrs ts(ls_id={id:1}, delta=553259, log_service_ts=1736326313146175, min_tx_service_ts=4611686018427387, timestamp={val:1736326313146175004, v:0}) [2025-01-08 00:51:53.699939] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=11, Other_Error_Cnt=0, Submit_Succ_Cnt=86, last_scn="{val:1736326312645069001, v:0}", last_lsn={lsn:18731075}, last_gts={val:1736326313646555000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-01-08 00:51:53.718485] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CBF-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:53.720714] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CBF-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:53.738709] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=48] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:53.738731] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:53.738781] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84991-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:53.738797] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84991-0-0] [lt=13] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:53.756547] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=197][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.758662] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84F0B-0-0] [lt=27] InfoPoolResizeTask(cost_ts=58) [2025-01-08 00:51:53.758950] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=82] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:53.774292] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=40] BlockGCTimerTask success(ret=0, cost_time_us=26, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:53.774872] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1226, avg_group_log_size=122) [2025-01-08 00:51:53.774922] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1217) [2025-01-08 00:51:53.777251] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2296, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:53.777305] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=51] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:53.777341] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1263, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:53.778074] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A962C-0-0] [lt=68] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:53.780512] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=33] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:53.782517] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=179] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:53.786696] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC89F7-0-0] [lt=98] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:53.786729] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC89F7-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9037, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:53.792727] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:53.794474] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46497][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6DF-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193818708306, cache_obj->added_lc()=false, cache_obj->get_object_id()=13687, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:53.795340] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4098, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:53.795382] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18732236}], scn_array=[cnt:1, {val:1736326313646555000, v:0}], curr_size=122, accum_size=1226, time_cost=18612) [2025-01-08 00:51:53.795409] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1226) [2025-01-08 00:51:53.795439] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1423, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:53.795584] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:53.795609] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8216, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:53.795619] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=73, avg_log_submit_to_flush_time=5621, avg_log_submit_to_slide_time=8142) [2025-01-08 00:51:53.795641] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=41, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:53.796163] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1226, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:53.802795] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=9281, avg_group_log_size=515) [2025-01-08 00:51:53.802825] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9825) [2025-01-08 00:51:53.805241] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=46] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4261, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:53.805270] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9281) [2025-01-08 00:51:53.805290] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=603, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:53.805383] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=169] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9825) [2025-01-08 00:51:53.805408] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=46, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:53.816577] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3BA8-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=27, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:53.816607] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3BA8-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=9802, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:53.816618] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3BA8-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=892, avg_log_gen_to_submit_time=1013, avg_log_submit_to_flush_time=5090, avg_log_submit_to_slide_time=8788) [2025-01-08 00:51:53.816810] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=127] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9281, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:53.816987] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326313143891004, v:0}, new_max_version={val:1736326313143891004, v:0}, last_min_version={val:1736326313143891002, v:0}, last_max_version={val:1736326313143891002, v:0}, total_time_us=16174, delta_us=673093) [2025-01-08 00:51:53.817016] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326313143891004, v:0}, min_version={val:1736326313143891004, v:0}, max_version={val:1736326313143891004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:53.817090] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1736326313644270001, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3, epoch_tstamp:1736326313817030}, version_delta=172818) [2025-01-08 00:51:53.818107] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=99, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:53.818246] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=122] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3651) [2025-01-08 00:51:53.819450] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC422-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:53.820754] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=2666, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:53.820787] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=461, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:53.820825] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=52] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3773) [2025-01-08 00:51:53.820847] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=35, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:53.821165] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:53.821249] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:53.821943] INFO [LIB] stat (utility.h:1210) [58631][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69EF-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=31, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:53.821964] INFO [LIB] stat (utility.h:1210) [58631][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69EF-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=6791, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:53.821973] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58631][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69EF-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=133, avg_log_submit_to_flush_time=3707, avg_log_submit_to_slide_time=6657) [2025-01-08 00:51:53.822555] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87172-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:53.822626] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:53.822639] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:53.823315] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4237, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:53.823608] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-01-08 00:51:53.823654] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=42] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:53.828496] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0BC-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:53.829052] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:202) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=6] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2025-01-08 00:51:53.829106] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=48] finish to resize log external storage handler(current_ts=1736326313829102, tenant_max_cpu=4, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:51:53.829122] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=15] snapshot_traffic_info once [2025-01-08 00:51:53.830055] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=34] gc stale ls task succ [2025-01-08 00:51:53.830302] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=46] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:53.830397] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=12] gc stale ls task succ [2025-01-08 00:51:53.831874] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1003}, {id:1002}]) [2025-01-08 00:51:53.832828] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:53.853353] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=80] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:53.855307] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=37] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:53.855405] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326304373137000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:53.856668] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4891-0-0] [lt=24] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:53.856720] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4891-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:53.856738] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4891-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:53.856696] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.866947] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=44] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:53.867683] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:53.869022] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=45] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:53.869145] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=46] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326305683208000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:53.869378] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61D-0-0] [lt=30] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:53.869442] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61D-0-0] [lt=46] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326313672968000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326313672968000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:53.869665] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:53.869755] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=81] temporary file module use no blocks [2025-01-08 00:51:53.869789] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:53.869805] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:53.869817] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193817777106, flush_io_finished_ret_:0, flush_io_finished_round_:3800, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:53.870948] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=4211, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:53.871074] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=122][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2246958}], scn_array=[cnt:1, {val:1736326313644270002, v:0}], curr_size=122, accum_size=1098, time_cost=14002) [2025-01-08 00:51:53.871097] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:53.871116] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=362, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:53.871127] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:263) [50687][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D2008458E-0-0] [lt=41] ObArchiveSender is running(thread_index=4) [2025-01-08 00:51:53.871275] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=94, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:53.871248] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=134] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:53.871805] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B33-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:53.871920] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B33-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:53.872377] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=55] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43580024}, readable_log_tail:{lsn:43580024}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23524744, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:53.873101] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=85] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:53.873176] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2351, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:53.873386] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=207] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:53.873408] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=651, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:53.873480] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=72, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:53.876399] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:53.876669] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48526, [delay/us]:prepare:8, schedule:78, submit:37, rt:2241, total:2365) [2025-01-08 00:51:53.876689] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:53.876744] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48516, [delay/us]: prepare:8, schedule:78, submit:37, device:2241, total:2365) [2025-01-08 00:51:53.876757] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:53.877022] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=209] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:72759, [delay/us]:prepare:12, schedule:237, submit:63, rt:1971, total:2284) [2025-01-08 00:51:53.877059] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:53.877100] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:72757, [delay/us]: prepare:12, schedule:237, submit:63, device:1971, total:2284) [2025-01-08 00:51:53.877112] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:53.877136] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:5, bw:20210, [delay/us]:prepare:5, schedule:119, submit:1, rt:1456, total:1583) [2025-01-08 00:51:53.877158] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:165724, [delay/us]:prepare:6, schedule:347, submit:216, rt:2277, total:2847) [2025-01-08 00:51:53.877167] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12126, [delay/us]:prepare:2, schedule:899, submit:26, rt:376, total:1304) [2025-01-08 00:51:53.877248] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=76] [IO STATUS TENANT](tenant_id=1002, ref_cnt=6, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49997, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:53.877286] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:5, avg_bw:20208, [delay/us]: prepare:5, schedule:119, submit:1, device:1456, total:1583) [2025-01-08 00:51:53.877302] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12125, [delay/us]: prepare:2, schedule:899, submit:26, device:376, total:1304) [2025-01-08 00:51:53.877309] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:41, avg_bw:165708, [delay/us]: prepare:6, schedule:347, submit:216, device:2277, total:2847) [2025-01-08 00:51:53.877320] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:53.877340] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4041, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:283, total:292) [2025-01-08 00:51:53.877365] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:53.877392] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4041, [delay/us]: prepare:0, schedule:0, submit:600000000, device:283, total:292) [2025-01-08 00:51:53.877410] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:53.877424] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:53.877432] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:53.877440] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:53.877448] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=233kB/s, net_out=205kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:53.877875] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=43] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8816, offset=2504550}, end_cursor:ObLogCursor{file_id=1, log_id=8817, offset=2504855}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cbc050, len:2202, log_data_len:305, seq:8816, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:53.880587] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E74-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:53.880649] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E74-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326313646555000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326313646555000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:53.891061] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=72] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:53.894771] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918544C-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:53.897232] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.897398] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.897431] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.897471] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:53.899910] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=18] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:53.900141] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8817, offset=2504855}, end_cursor:ObLogCursor{file_id=1, log_id=8818, offset=2505152}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cb8050, len:1897, log_data_len:297, seq:8817, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:53.904961] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E74-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:53.905051] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E74-0-0] [lt=55] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326313646555002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326313646555002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:53.919921] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=140] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:53.921726] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=52] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:53.921897] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=51] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326305812845000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:53.921957] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=56] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326305812845000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326305812845000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:53.924968] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=88] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18367, miss_cnt:0, cache_read_size:2246958, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:53.953531] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58635][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4893-0-0] [lt=65] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=28, avg_log_submit_to_flush_time=4658, avg_log_submit_to_slide_time=32343) [2025-01-08 00:51:53.953841] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:53.958958] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:53.963101] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3773) [2025-01-08 00:51:53.966080] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B35-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:53.966118] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B35-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=42870, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:53.966131] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B35-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=24, avg_log_submit_to_flush_time=3186, avg_log_submit_to_slide_time=42845) [2025-01-08 00:51:53.966315] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:53.969891] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:53.974233] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58635][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=149] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:53.985414] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=52] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:54.015643] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C7-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:54.015738] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C7-0-0] [lt=71] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326313644270003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326313644270003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:54.020572] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get number(ret=0, service_type_=0, range=1, base_id=1736326314016994000, start_id=1736326314016994000, end_id=1736326314016994001) [2025-01-08 00:51:54.023444] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=103] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=17365377, this={dio_aligned_buf:{buf_write_offset:3019, buf_padding_size:1077, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:71124, truncate_used_ts:17518}, log_block_size:67108864, io_fd:{first_id:-1, second_id:152, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=17362944, buf_len=586, total_write_size=53752, total_write_size_after_dio=569344, ob_pwrite_used_ts=707587, count=126, each_pwrite_cost=5615) [2025-01-08 00:51:54.029402] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58638][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:54.029475] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:51:54.048006] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=54] PNIO [ratelimit] time: 1736326314047999, bytes: 0, bw: 0.000000 MB/s, add_ts: 1002572, add_bytes: 0 [2025-01-08 00:51:54.055663] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5307481385, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:54.055717] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:54.055786] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=67] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a02b0) [2025-01-08 00:51:54.055798] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5307481385, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:54.059246] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.066951] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:51:54.068124] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=111] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:54.073240] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:54.148226] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:54.152196] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:54.153031] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4897-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=24, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:54.153085] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4897-0-0] [lt=54] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30298, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:54.154740] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:54.157884] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:54.159366] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.161166] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:292K/70:134753 disk:292K/70:134753 t1m2:44K/10:39506 t1m2g0:44K/10:39506 t1002m2:168K/42:184751 t1002m2g0:168K/42:184751 t1001m2:80K/18:71007 t1001m2g0:80K/18:71007 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:54.170508] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:19824, max_lsn:18732724, max_scn:{val:1736326313646555003, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:18732724}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:18732602}, last_submit_end_lsn:{lsn:18732724}, last_submit_log_id:19824, last_submit_log_pid:1, max_flushed_lsn:{lsn:18732602}, max_flushed_end_lsn:{lsn:18732724}, max_flushed_log_pid:1, committed_end_lsn:{lsn:18732724}, last_slide_log_id:19824, last_slide_scn:{val:1736326313646555003, v:0}, last_slide_lsn:{lsn:18732602}, last_slide_end_lsn:{lsn:18732724}, last_slide_log_pid:1, last_slide_log_accum_checksum:505376677, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8fc454530}, [StateMgr]={this:0x2ad8fc458830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:19825, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324399554770, palf_role_change_cb:0x2ad8fc45f2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8fc455c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324399758968, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/meta", dir_fd:175, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8fc45aa38, flashback_version:0}, log_storage:{log_tail:{lsn:18732724}, readable_log_tail:{lsn:18732724}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48372044, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324399554279, this:0x2ad8fc458f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324399555093, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8fc4589b0}) [2025-01-08 00:51:54.170882] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=368] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326314078470, last_flushed_end_lsn:{lsn:18732724}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326314082419, last_flushed_end_lsn:{lsn:18732724}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326314079822, last_flushed_end_lsn:{lsn:18732724}}]) [2025-01-08 00:51:54.170908] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=19825, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326314078470, last_flushed_end_lsn:{lsn:18732724}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326314082419, last_flushed_end_lsn:{lsn:18732724}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326314079822, last_flushed_end_lsn:{lsn:18732724}}]) [2025-01-08 00:51:54.170950] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1002, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18464, max_lsn:2247446, max_scn:{val:1736326314144466000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2247446}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2247324}, last_submit_end_lsn:{lsn:2247446}, last_submit_log_id:18464, last_submit_log_pid:1, max_flushed_lsn:{lsn:2247324}, max_flushed_end_lsn:{lsn:2247446}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2247446}, last_slide_log_id:18464, last_slide_scn:{val:1736326314144466000, v:0}, last_slide_lsn:{lsn:2247324}, last_slide_end_lsn:{lsn:2247446}, last_slide_log_pid:1, last_slide_log_accum_checksum:2122840886, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad92435e530}, [StateMgr]={this:0x2ad924362830, self:"192.168.159.137:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.138:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18465, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad9243692f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326313162481, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326313162481, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad92435fc70}, [ModeMgr]={palf_id:1002, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436803442001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/meta", dir_fd:184, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924364a38, flashback_version:0}, log_storage:{log_tail:{lsn:2247446}, readable_log_tail:{lsn:2247446}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64857322, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:-1, this:0x2ad924362f70}, [Reconfirm]={palf_id:1002, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9243629b0}) [2025-01-08 00:51:54.171209] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=213][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:54.171228] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:54.171238] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:54.171247] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] current log_task status(palf_id=1002, self="192.168.159.137:2882", log_id=18465, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:51:54.171296] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326310124991, last_first_register_time_us=1736324454612308, register_time_us=1736326310124991, curr_time_us=1736326314171295) [2025-01-08 00:51:54.171361] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326314171314, last_submit_register_req_time_us=1736326314171314, register_parent_reason=1) [2025-01-08 00:51:54.171394] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:54.171508] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=193] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:54.171710] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=56] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:54.172695] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:51:54.175991] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=56, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:54.176069] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:54.176197] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=67, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:54.176231] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:54.178887] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:54.204996] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:54.249604] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF6D-0-0] [lt=19] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326314366988, keepalive_ts:-1}) [2025-01-08 00:51:54.249726] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF6D-0-0] [lt=120] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326314366988, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:54.259509] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.262983] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A83-0-0] [lt=21] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326314262963}) [2025-01-08 00:51:54.264734] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A5-0-0] [lt=105] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326314264706, sql_renew_time:1736326314264706, is_mark_fail:false}]) [2025-01-08 00:51:54.264779] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A5-0-0] [lt=43] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326314262963}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326314264706, sql_renew_time:1736326314264706, is_mark_fail:false}]) [2025-01-08 00:51:54.267110] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58611][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B3B-0-0] [lt=14][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18362, log_task={header:{begin_lsn:{lsn:2234925}, end_lsn:{lsn:2235047}, log_id:18362, min_scn:{val:1736326313672968004, v:0}, max_scn:{val:1736326313672968004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2234803}, prev_proposal_id:1, committed_end_lsn:{lsn:2234925}, data_checksum:940009244, accum_checksum:2907744400, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326314165713, freeze_ts:1736326314165713, submit_ts:1736326314165724, flushed_ts:1736326314167595, gen_to_freeze cost time:0, gen_to_submit cost time:11, submit_to_flush cost time:1871}, fs_cb_begin_ts=1736326314267090, log_life_time=101377) [2025-01-08 00:51:54.268649] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:54.273461] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:54.273540] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4015] clock out of order(cur_ts=1736326314273538, cur_ts_=1736326314250578, delta=22960) [2025-01-08 00:51:54.274060] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43585175}], scn_array=[cnt:1, {val:1736326314250578000, v:0}], curr_size=589, accum_size=6329, time_cost=19679) [2025-01-08 00:51:54.274552] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2247446}], scn_array=[cnt:1, {val:1736326314144466001, v:0}], curr_size=122, accum_size=488, time_cost=20232) [2025-01-08 00:51:54.289589] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18732846}], scn_array=[cnt:1, {val:1736326314147983001, v:0}], curr_size=122, accum_size=610, time_cost=13260) [2025-01-08 00:51:54.289776] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2213566}], scn_array=[cnt:1, {val:1736326314147983001, v:0}], curr_size=122, accum_size=610, time_cost=13167) [2025-01-08 00:51:54.289779] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1226) [2025-01-08 00:51:54.298272] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=61] ls blacklist refresh finish(cost_time=36985) [2025-01-08 00:51:54.301069] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=96] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:54.319556] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=52] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.319826] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.335943] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46501][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6E1-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193819249773, cache_obj->added_lc()=false, cache_obj->get_object_id()=13688, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:54.347037] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6E6-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326314345940000, start_id=1736326314345940000, end_id=1736326314345940001) [2025-01-08 00:51:54.350827] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=10] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:54.353581] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=109, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:54.353703] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=119] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:54.354568] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=35] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:54.359641] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.362515] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCC-0-0] [lt=68] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326314479406, keepalive_ts:-1}) [2025-01-08 00:51:54.362578] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58609][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCC-0-0] [lt=60] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326314479406, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:54.366302] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=159] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18273, miss_cnt:0, cache_read_size:2235169, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:54.368177] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=157] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=93, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:54.376519] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:54.376612] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=88] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:54.404644] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AF2F-0-0] [lt=61] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.817513704299926758e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.517570905390110738e+02) [2025-01-08 00:51:54.404939] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=68] get number(ret=0, service_type_=0, range=1, base_id=1736326314404494000, start_id=1736326314404494000, end_id=1736326314404494001) [2025-01-08 00:51:54.413667] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:54.413696] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:54.413708] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:54.413720] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:54.417006] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:54.418071] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=1058] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:54.421034] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4567, avg_group_log_size=415) [2025-01-08 00:51:54.458887] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=75] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.459307] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.459840] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.469295] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:54.473573] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:54.503064] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=100] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326314144466002, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=358576, in_cluster_service=true, cluster_version={val:1736326313672968003, v:0}, min_cluster_version={val:1736326313672968003, v:0}, max_cluster_version={val:1736326313672968003, v:0}, get_cluster_version_err=0, cluster_version_delta=830074, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18673, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=542420569, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:54.519008] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=58] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.540800] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=84] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:54.541739] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.544604] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.559999] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.573210] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] LogLoopThread round_cost_time(us)(round_cost_time=109) [2025-01-08 00:51:54.573299] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30] LogLoopThread round_cost_time(us)(round_cost_time=7) [2025-01-08 00:51:54.585906] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=132] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:54.606780] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:112) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AF32-0-0] [lt=90] adaptive update(Tenant ID=1001, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=390, Holding Memory(MB)=102, Tenant Remain Memory(MB)=480, Usable Remain Memory(MB)=288, Last Update Limit Timestamp=1736326314605633, Is Updated=true) [2025-01-08 00:51:54.611705] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=41] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324400232263, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:54.632633] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [50591][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=23] XA scheduler heartbeat task statistics(avg_time=0) [2025-01-08 00:51:54.646851] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=96] current monitor number(seq_=-1) [2025-01-08 00:51:54.647397] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1736326314647394, bytes: 883867888, bw: 0.248890 MB/s, add_ts: 1002789, add_bytes: 261708 [2025-01-08 00:51:54.648009] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=34] gc stale ls task succ [2025-01-08 00:51:54.649522] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=89] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=501, get_gts_with_stc_cnt=3, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=3, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:54.651378] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:54.651442] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=54] temporary file module use no blocks [2025-01-08 00:51:54.651457] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:54.651471] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:54.651486] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193818565006, flush_io_finished_ret_:0, flush_io_finished_round_:1977, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:54.655363] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=30] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:54.658490] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=33] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:54.659907] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=115] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:54.660159] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.664933] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:54.664964] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] try gc retain ctx [2025-01-08 00:51:54.664973] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:54.665041] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=46] get_max_applied_scn(scn={val:1736326314617680000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84469201}, palf_committed_end_scn_:{val:1736326314617680000, v:0}, last_check_scn_:{val:1736326314617680000, v:0}, max_applied_cb_scn_:{val:1736326314617680000, v:0}}) [2025-01-08 00:51:54.665072] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84469201}, readable_log_tail:{lsn:84469201}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49740335, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:54.665155] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=80] finish update min start scn(max_decided_scn={val:1736326314617680000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326304616286001, v:0}, keep_alive_scn_:{val:1736326309543479001, v:0}, update_ts_:1736326314664924}) [2025-01-08 00:51:54.666127] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329254996000, latest_log_ts:{val:1736326312592647001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:54.668673] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=133] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:54.668695] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] try gc retain ctx [2025-01-08 00:51:54.668702] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-01-08 00:51:54.668724] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1736326314453572000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43589693}, palf_committed_end_scn_:{val:1736326314657020000, v:0}, last_check_scn_:{val:1736326314657020000, v:0}, max_applied_cb_scn_:{val:1736326314453572000, v:0}}) [2025-01-08 00:51:54.668746] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] finish update min start scn(max_decided_scn={val:1736326314453572000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326304600320001, v:0}, keep_alive_scn_:{val:1736326309628490001, v:0}, update_ts_:1736326314668316}) [2025-01-08 00:51:54.669848] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329284252000, latest_log_ts:{val:1736326312645069001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:54.671013] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:54.671977] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD8-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:54.672024] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD8-0-0] [lt=43] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:54.672999] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE941-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1736326314672545000, start_id=1736326314672545000, end_id=1736326314672545001) [2025-01-08 00:51:54.674868] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=87] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1509888, replace_num=15728) [2025-01-08 00:51:54.674889] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:54.674973] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=151] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:54.674937] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=65] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:54.675148] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=67] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:54.679327] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2235526}], scn_array=[cnt:1, {val:1736326314165163004, v:0}], curr_size=122, accum_size=967, time_cost=11584) [2025-01-08 00:51:54.686507] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:54.687719] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58636][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=306] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:51:54.717547] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=42] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:54.734941] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC0-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:54.736855] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC0-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:54.752047] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=22] ====== tenant freeze timer task ====== [2025-01-08 00:51:54.760359] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=119][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.765401] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=83] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:54.767397] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085173-0-0] [lt=16] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:54.767550] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085175-0-0] [lt=34] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:54.767760] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=57][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:54.768024] WDIAG read_multi_row (ob_table_access_helper.h:238) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=124][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:51:54.769890] INFO delete_row (ob_table_access_helper.h:339) [46330][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=46] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326314768072) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326314768072) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:51:54.778384] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE45-0-0] [lt=15] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:54.778434] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE45-0-0] [lt=42] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:54.778468] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE45-0-0] [lt=31] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:54.778828] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:54.778869] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:54.780929] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84469314}], scn_array=[cnt:1, {val:1736326314648302001, v:0}], curr_size=122, accum_size=3387, time_cost=10967) [2025-01-08 00:51:54.781234] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2235648}], scn_array=[cnt:1, {val:1736326314664924000, v:0}], curr_size=122, accum_size=1089, time_cost=10272) [2025-01-08 00:51:54.781360] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=27] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:54.782005] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3535, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:54.782031] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:54.782047] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=416, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:54.785255] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=101] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:54.785530] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=55] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:54.801120] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:54.803362] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1736326314802629000, start_id=1736326314802629000, end_id=1736326314802629001) [2025-01-08 00:51:54.809259] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=65] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=20, avg_log_batch_cnt=1, total_group_log_size=11090, avg_group_log_size=554) [2025-01-08 00:51:54.809313] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=48] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11090) [2025-01-08 00:51:54.817500] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4812, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:54.817531] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18733578}], scn_array=[cnt:1, {val:1736326314802629000, v:0}], curr_size=545, accum_size=545, time_cost=13207) [2025-01-08 00:51:54.817578] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=248, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:54.817678] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:54.822672] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58609][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F24-0-0] [lt=24] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11090) [2025-01-08 00:51:54.826266] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:54.826301] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:54.828326] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=88, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:54.828379] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4095) [2025-01-08 00:51:54.829772] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC425-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:54.830550] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50606][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=71][errcode=0] task scheduled too delay(scheduled_delay=1260315, expected_run_ts_us_=1736326313570231, ObClockGenerator::getRealClock()=1736326314830548, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfedb0, caller:ob_multi_version_garbage_collector.cpp:start:116, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE6$_1073, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dff090, time_wheel:0x2ad8dd5c3990, thread_pool:0x2ad8dd5c3170, is_running:1, time_interval:60.00s, expected_run_time:00:51:53.570, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:54.830772] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326314823702000, start_id=1736326314823702000, end_id=1736326314823702001) [2025-01-08 00:51:54.830797] INFO [MVCC] study (ob_multi_version_garbage_collector.cpp:329) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=23] study min unallocated gts succeed(ret=0, min_unallocated_GTS={val:1736326314823702000, v:0}, this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}) [2025-01-08 00:51:54.830831] INFO [STORAGE.TRANS] get_cluster_version (ob_weak_read_service.cpp:133) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] get_cluster_version(ret=0, tenant_id=1001, version={val:1736326314422615000, v:0}) [2025-01-08 00:51:54.830843] INFO [MVCC] study (ob_multi_version_garbage_collector.cpp:347) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10] study min unallocated wrs succeed(ret=0, min_unallocated_WRS={val:1736326314422615000, v:0}, this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}, is_primary=true) [2025-01-08 00:51:54.830857] INFO [MVCC] study (ob_multi_version_garbage_collector.cpp:363) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] study max committed txn version succeed(ret=0, max_committed_txn_version={val:1736326314657020000, v:0}, this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}) [2025-01-08 00:51:54.831091] INFO [MVCC] study (ob_multi_version_garbage_collector.cpp:374) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] study min active txn version succeed(ret=0, min_active_txn_version={val:4611686018427387903, v:0}, this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}) [2025-01-08 00:51:54.834613] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6E7-0-0] [lt=119][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193819748447, cache_obj->added_lc()=false, cache_obj->get_object_id()=13689, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:54.834902] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:51:54.834948] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=67] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:54.834999] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:51:54.835095] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1}, {id:1002}]) [2025-01-08 00:51:54.835131] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=76] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:54.835289] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922105c48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:67307} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad9220f8e70 trans_version={val:1736325654684017000, v:0} scn={val:1736325654684017000, v:0} tx_id={txid:18213} prev=0x2ad9220f8af0 next=(nil) modify_count=4 acc_checksum=2464639132 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326314823702001, v:0}, *this={this=0x2ad9220a49a8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220f8e70 latest_compact_node=(nil) max_trans_version={val:1736325654684017000, v:0} max_trans_id=18213 max_elr_trans_version={val:1736325654684017000, v:0} max_elr_trans_id=18213 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325654684017000, v:0} min_modify_scn={val:1736324394708167000, v:0}}) [2025-01-08 00:51:54.835335] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=45][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922105c48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:67307} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad90bbc8908, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326344732525, tablet_id:{id:444}, table_iter:0x2ad90bbc6df8, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326344732525, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326314823702001, v:0}, tx_id:{txid:67307}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:67307}, tx_desc:{this:0x2ad8b6018d60, tx_id:{txid:67307}, state:1, addr:"192.168.159.137:2882", tenant_id:1001, session_id:3221587401, assoc_session_id:3221587401, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326314823702, active_ts:1736326314823702, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326314930799, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad8fb6e30d0, mem_ctx:0x2ad8fb6e4650, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:54.835419] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=83][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:67307}, data:0x2ad90bbc57c8, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:51:54.835436] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:67307} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:54.835465] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:54.835476] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad962ef1db0, local_datums:0x2ad962ef1db0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1a14, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1a1c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad962ef1a24, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1a33, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1a3b, hex: 02AD35F92D2B0600, int: 1736326314831106}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1a43, hex: F0D53A7995AB1818, int: 1736326314823702000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1a4b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:54.835561] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=50][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:54.835598] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922105de0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:67307} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad9220f9008 trans_version={val:1736325654684017000, v:0} scn={val:1736325654684017000, v:0} tx_id={txid:18213} prev=0x2ad9220f8b98 next=(nil) modify_count=4 acc_checksum=2928118392 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326314823702001, v:0}, *this={this=0x2ad9220b1160 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220f9008 latest_compact_node=(nil) max_trans_version={val:1736325654684017000, v:0} max_trans_id=18213 max_elr_trans_version={val:1736325654684017000, v:0} max_elr_trans_id=18213 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325654684017000, v:0} min_modify_scn={val:1736324394708167000, v:0}}) [2025-01-08 00:51:54.835629] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=31][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922105de0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:67307} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad90bbc8908, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326344732525, tablet_id:{id:444}, table_iter:0x2ad90bbc6df8, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326344732525, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326314823702001, v:0}, tx_id:{txid:67307}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:67307}, tx_desc:{this:0x2ad8b6018d60, tx_id:{txid:67307}, state:1, addr:"192.168.159.137:2882", tenant_id:1001, session_id:3221587401, assoc_session_id:3221587401, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326314823702, active_ts:1736326314823702, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326314930799, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad8fb6e30d0, mem_ctx:0x2ad8fb6e4650, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:54.835693] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=63][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:67307}, data:0x2ad90bbc57c8, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:51:54.835706] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=12][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":2},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:67307} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:54.835727] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=20][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:54.835736] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad962ef1db0, local_datums:0x2ad962ef1db0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1ad4, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1adc, hex: 0200000000000000, int: 2}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad962ef1ae4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1af3, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1afb, hex: 02AD35F92D2B0600, int: 1736326314831106}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1b03, hex: D8BB526195AB1818, int: 1736326314422615000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1b0b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:54.835794] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=37][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:54.835827] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922105f78 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:67307} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad9220f91a0 trans_version={val:1736325654684017000, v:0} scn={val:1736325654684017000, v:0} tx_id={txid:18213} prev=0x2ad9220f8c40 next=(nil) modify_count=4 acc_checksum=3561049257 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326314823702001, v:0}, *this={this=0x2ad9220b1318 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220f91a0 latest_compact_node=(nil) max_trans_version={val:1736325654684017000, v:0} max_trans_id=18213 max_elr_trans_version={val:1736325654684017000, v:0} max_elr_trans_id=18213 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325654684017000, v:0} min_modify_scn={val:1736324394708167000, v:0}}) [2025-01-08 00:51:54.835857] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=30][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922105f78 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:67307} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad90bbc8908, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326344732525, tablet_id:{id:444}, table_iter:0x2ad90bbc6df8, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326344732525, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326314823702001, v:0}, tx_id:{txid:67307}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:67307}, tx_desc:{this:0x2ad8b6018d60, tx_id:{txid:67307}, state:1, addr:"192.168.159.137:2882", tenant_id:1001, session_id:3221587401, assoc_session_id:3221587401, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326314823702, active_ts:1736326314823702, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326314930799, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad8fb6e30d0, mem_ctx:0x2ad8fb6e4650, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:54.835918] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=61][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:67307}, data:0x2ad90bbc57c8, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:51:54.835931] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=11][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:67307} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:54.835952] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=20][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:54.835960] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad962ef1db0, local_datums:0x2ad962ef1db0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1b94, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1b9c, hex: 0300000000000000, int: 3}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad962ef1ba4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1bb3, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1bbb, hex: 02AD35F92D2B0600, int: 1736326314831106}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1bc3, hex: 60784B6F95AB1818, int: 1736326314657020000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1bcb, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:54.836018] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=37][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:54.836049] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=9][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922106110 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:67307} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad9220f9338 trans_version={val:1736325654684017000, v:0} scn={val:1736325654684017000, v:0} tx_id={txid:18213} prev=0x2ad9220f8ce8 next=(nil) modify_count=4 acc_checksum=3204960563 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326314823702001, v:0}, *this={this=0x2ad9220b14d0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220f9338 latest_compact_node=(nil) max_trans_version={val:1736325654684017000, v:0} max_trans_id=18213 max_elr_trans_version={val:1736325654684017000, v:0} max_elr_trans_id=18213 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325654684017000, v:0} min_modify_scn={val:1736324394708167000, v:0}}) [2025-01-08 00:51:54.836079] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=29][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922106110 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:67307} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad90bbc8908, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326344732525, tablet_id:{id:444}, table_iter:0x2ad90bbc6df8, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326344732525, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326314823702001, v:0}, tx_id:{txid:67307}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:67307}, tx_desc:{this:0x2ad8b6018d60, tx_id:{txid:67307}, state:1, addr:"192.168.159.137:2882", tenant_id:1001, session_id:3221587401, assoc_session_id:3221587401, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326314823702, active_ts:1736326314823702, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326314930799, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad8fb6e30d0, mem_ctx:0x2ad8fb6e4650, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:51:54.836141] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=61][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:67307}, data:0x2ad90bbc57c8, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:51:54.836186] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=44][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":4},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:67307} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:51:54.836208] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=21][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:51:54.836217] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad962ef1db0, local_datums:0x2ad962ef1db0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1c54, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1c5c, hex: 0400000000000000, int: 4}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad962ef1c64, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1c73, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1c7b, hex: 02AD35F92D2B0600, int: 1736326314831106}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1c83, hex: FFFFFFFFFFFFFF3F, int: 4611686018427387903}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad962ef1c8b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:51:54.836326] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=84][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:51:54.836386] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [50605][T1001_Occam][T1001][YB42C0A89F89-00062B2D21C845BA-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:67307} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:51:54.837077] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=267] ====== checkpoint timer task ====== [2025-01-08 00:51:54.837993] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=15] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:2, active_tx_count:2, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:54.838032] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=39] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1736326314802629000, v:0}) [2025-01-08 00:51:54.838133] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=20] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329284252000, latest_log_ts:{val:1736326312645069001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:54.838208] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=71] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:54.838297] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=82] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:54.838317] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=17] get rec log scn(service_type_=0, rec_log_ts={val:1736326312645069001, v:0}) [2025-01-08 00:51:54.838328] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:54.838339] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:54.838353] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=8] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:54.838370] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:54.838383] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:54.838404] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:54.838424] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:54.838433] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:54.838558] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A16-0-0] [lt=117] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=48, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:54.838574] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A16-0-0] [lt=15] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=8749, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:54.838582] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A16-0-0] [lt=6] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=91, avg_log_submit_to_flush_time=5155, avg_log_submit_to_slide_time=8658) [2025-01-08 00:51:54.838943] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1765, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:54.840478] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4042, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:54.840511] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1376, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:54.840584] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4095) [2025-01-08 00:51:54.840611] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=45, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:54.842350] INFO [LIB] stat (utility.h:1210) [58631][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69FA-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:54.842371] INFO [LIB] stat (utility.h:1210) [58631][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69FA-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7461, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:54.842381] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58631][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA69FA-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=26, avg_log_gen_to_submit_time=130, avg_log_submit_to_flush_time=5520, avg_log_submit_to_slide_time=7331) [2025-01-08 00:51:54.842533] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=97] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4095, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:54.842963] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=63] [PALF STAT WRITE LOG COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=4647, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:54.842987] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43589936}], scn_array=[cnt:1, {val:1736326314802629000, v:0}], curr_size=589, accum_size=11090, time_cost=26263) [2025-01-08 00:51:54.843012] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=11090) [2025-01-08 00:51:54.843044] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=835, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:54.843120] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326314617680000, v:0}, new_max_version={val:1736326314617680000, v:0}, last_min_version={val:1736326314412287000, v:0}, last_max_version={val:1736326314412287000, v:0}, total_time_us=16698, delta_us=225438) [2025-01-08 00:51:54.843153] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326314617680000, v:0}, min_version={val:1736326314617680000, v:0}, max_version={val:1736326314617680000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:54.843807] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=24, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:54.843849] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT LOG LIFE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=7682, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:54.843860] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=20, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=139, avg_log_submit_to_flush_time=5687, avg_log_submit_to_slide_time=7542) [2025-01-08 00:51:54.843884] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=61, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:54.843961] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=11090, hit_cnt=21, read_cnt=21, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:54.860637] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:54.861014] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=374][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.863090] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43590525}], scn_array=[cnt:1, {val:1736326314836230000, v:0}], curr_size=1864, accum_size=1864, time_cost=18057) [2025-01-08 00:51:54.867253] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D0BF-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:54.870037] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:54.870085] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=41] temporary file module use no blocks [2025-01-08 00:51:54.870098] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:54.870404] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=302] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:54.870470] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=65] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193818783687, flush_io_finished_ret_:0, flush_io_finished_round_:3801, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:54.871477] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B47-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:51:54.871557] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B47-0-0] [lt=39][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18368, log_task={header:{begin_lsn:{lsn:2235648}, end_lsn:{lsn:2235770}, log_id:18368, min_scn:{val:1736326314664924000, v:0}, max_scn:{val:1736326314664924000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2235526}, prev_proposal_id:1, committed_end_lsn:{lsn:2235648}, data_checksum:940009244, accum_checksum:3458653205, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326314770659, freeze_ts:1736326314770659, submit_ts:1736326314770667, flushed_ts:1736326314781401, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:10734}, fs_cb_begin_ts=1736326314871524, log_life_time=100865) [2025-01-08 00:51:54.872746] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6724874240, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:54.872961] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:54.879605] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48A5-0-0] [lt=60] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:54.879652] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48A5-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:54.879671] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48A5-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:54.880578] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:54.880994] INFO [MVCC] report (ob_multi_version_garbage_collector.cpp:799) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=23] report multi version snapshot success(ret=0, ret="OB_SUCCESS", sql= insert into __all_reserved_snapshot (tenant_id, snapshot_type, svr_ip, svr_port, create_time, status, snapshot_version) values (1001, 1, '192.168.159.137', 2882, '1736326314831106', 0, 1736326314823702000), (1001, 2, '192.168.159.137', 2882, '1736326314831106', 0, 1736326314422615000), (1001, 3, '192.168.159.137', 2882, '1736326314831106', 0, 1736326314657020000), (1001, 4, '192.168.159.137', 2882, '1736326314831106', 0, 4611686018427387903) on duplicate key update create_time = VALUES(create_time), snapshot_version = VALUES(snapshot_version) , affected_rows=8) [2025-01-08 00:51:54.881021] INFO [MVCC] study (ob_multi_version_garbage_collector.cpp:393) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=25] study multi version garabage collector end(ret=0, this={this:0x2ad8f95bf030, last_study_timestamp:1736325654689702, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}, min_unallocated_GTS={val:1736326314823702000, v:0}, min_unallocated_GTS={val:1736326314823702000, v:0}, max_committed_txn_version={val:1736326314657020000, v:0}, min_active_txn_version={val:4611686018427387903, v:0}) [2025-01-08 00:51:54.881048] INFO [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:193) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=25] repeat study successfully(ret=0, this={this:0x2ad8f95bf030, last_study_timestamp:1736326314881047, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}, current_timestamp=1736326314830766, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:51:54.882426] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:54.882798] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736326314823702000, snapshot_type=1, create_time=1736326314831106, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:54.882823] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325662041557001, snapshot_type=1, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:54.882834] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325663340298000, snapshot_type=1, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:54.882843] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736326314422615000, snapshot_type=2, create_time=1736326314831106, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:54.882853] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325661073141000, snapshot_type=2, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:54.882862] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325662946686001, snapshot_type=2, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:54.882871] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736326314657020000, snapshot_type=3, create_time=1736326314831106, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:54.882880] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325661436394000, snapshot_type=3, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:54.882889] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325663119496000, snapshot_type=3, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:54.882898] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736326314831106, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:54.882907] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:54.882916] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:54.883128] INFO [MVCC] decide_reserved_snapshot_version_ (ob_multi_version_garbage_collector.cpp:620) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update global reserved snapshot(ret=0, this={this:0x2ad8f95bf030, last_study_timestamp:1736326314881047, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325654363217000, v:0}, is_inited:true}, global_reserved_snapshot_={val:1736325654363217000, v:0}, reserved_snapshot={val:1736325661073141000, v:0}) [2025-01-08 00:51:54.883151] INFO [MVCC] refresh_ (ob_multi_version_garbage_collector.cpp:554) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] multi-version garbage collector refresh successfully(this={this:0x2ad8f95bf030, last_study_timestamp:1736326314881047, last_refresh_timestamp:1736326254217720, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325661073141000, v:0}, is_inited:true}, collector={reserved_snapshot_version:{val:1736325661073141000, v:0}, reserved_snapshot_type:2, reserved_status:-1, reserved_create_time:1736325662486709, reserved_addr:"192.168.159.138:2882", is_this_server_disabled:false, status:0}) [2025-01-08 00:51:54.883169] INFO [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:232) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] repeat refresh successfully(ret=0, this={this:0x2ad8f95bf030, last_study_timestamp:1736326314881047, last_refresh_timestamp:1736326314883168, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325661073141000, v:0}, is_inited:true}, current_timestamp=1736326314881061, GARBAGE_COLLECT_RETRY_INTERVAL=60000000) [2025-01-08 00:51:54.883882] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58638][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D62-0-0] [lt=47] init point thread id with(&point=0x5572d50e6d00, idx_=1295, point=[thread id=58638, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58638) [2025-01-08 00:51:54.884745] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=67] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736326314823702000, snapshot_type=1, create_time=1736326314831106, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:54.884769] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=21] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325662041557001, snapshot_type=1, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:54.884780] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325663340298000, snapshot_type=1, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:54.884790] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736326314422615000, snapshot_type=2, create_time=1736326314831106, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:54.884800] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325661073141000, snapshot_type=2, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:54.884809] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325662946686001, snapshot_type=2, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:54.884819] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736326314657020000, snapshot_type=3, create_time=1736326314831106, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:54.884828] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325661436394000, snapshot_type=3, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:54.884837] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=1736325663119496000, snapshot_type=3, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:54.884847] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736326314831106, addr="192.168.159.137:2882", gc_status=0) [2025-01-08 00:51:54.884856] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325662486709, addr="192.168.159.138:2882", gc_status=0) [2025-01-08 00:51:54.884865] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:709) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1001 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1736325663421538, addr="192.168.159.139:2882", gc_status=0) [2025-01-08 00:51:54.884983] INFO [MVCC] reclaim (ob_multi_version_garbage_collector.cpp:981) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=21] skip all alivavle snapshots info [2025-01-08 00:51:54.885000] INFO [MVCC] monitor_ (ob_multi_version_garbage_collector.cpp:1061) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] garbage collector monitor server status monitor(snapshot_servers=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882", "192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882", "192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882", "192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], lost_servers=[]) [2025-01-08 00:51:54.885019] INFO [MVCC] reclaim (ob_multi_version_garbage_collector.cpp:1011) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] reclaim multi version garabage collector end(this={this:0x2ad8f95bf030, last_study_timestamp:1736326314881047, last_refresh_timestamp:1736326314883168, last_reclaim_timestamp:1736325654693427, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325661073141000, v:0}, is_inited:true}, ret=0, role=1, reclaimable_servers=[], snapshot_servers=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882", "192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882", "192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882", "192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], is_this_server_disabled=false) [2025-01-08 00:51:54.885048] INFO [MVCC] repeat_reclaim (ob_multi_version_garbage_collector.cpp:263) [50605][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=27] repeat reclaim successfully(ret=0, this={this:0x2ad8f95bf030, last_study_timestamp:1736326314881047, last_refresh_timestamp:1736326314883168, last_reclaim_timestamp:1736326314885047, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1736325661073141000, v:0}, is_inited:true}, current_timestamp=1736326314883236, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-01-08 00:51:54.885848] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4799, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:54.885940] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=89][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2235770}], scn_array=[cnt:1, {val:1736326314664924001, v:0}], curr_size=122, accum_size=1211, time_cost=12690) [2025-01-08 00:51:54.885966] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:51:54.885985] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=791, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:54.886071] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=99, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:54.888677] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A18-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=43, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:54.888710] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A18-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6216, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:54.892149] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87178-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:54.892426] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:54.892450] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:54.893567] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:54.893635] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48322, [delay/us]:prepare:41, schedule:278, submit:79, rt:3549, total:3948) [2025-01-08 00:51:54.893660] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:54.893738] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=56] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48330, [delay/us]: prepare:41, schedule:278, submit:79, device:3549, total:3948) [2025-01-08 00:51:54.893781] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:54.893852] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:21, bw:96675, [delay/us]:prepare:13, schedule:139, submit:763, rt:4322, total:5238) [2025-01-08 00:51:54.893946] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=87] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:54.894031] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:21, avg_bw:96667, [delay/us]: prepare:13, schedule:139, submit:763, device:4322, total:5238) [2025-01-08 00:51:54.894052] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:54.894089] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:2, bw:8055, [delay/us]:prepare:5, schedule:14, submit:1, rt:1701, total:1722) [2025-01-08 00:51:54.894117] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:38, bw:157081, [delay/us]:prepare:10, schedule:255, submit:301, rt:3761, total:4329) [2025-01-08 00:51:54.894128] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1002, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:54.894161] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:2, avg_bw:8056, [delay/us]: prepare:5, schedule:14, submit:1, device:1701, total:1722) [2025-01-08 00:51:54.894177] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:39, avg_bw:157094, [delay/us]: prepare:10, schedule:255, submit:301, device:3761, total:4329) [2025-01-08 00:51:54.894235] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:54.894284] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4027, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:281, total:288) [2025-01-08 00:51:54.894312] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:54.894343] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4027, [delay/us]: prepare:0, schedule:0, submit:600000000, device:281, total:288) [2025-01-08 00:51:54.894362] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:54.894378] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:54.894387] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:54.894394] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:54.894403] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=225kB/s, net_out=247kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:54.894675] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5396, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:54.894700] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:54.894720] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=848, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:54.894792] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=127, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:54.898700] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918544D-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:54.898967] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:54.898993] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-01-08 00:51:54.899003] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:54.899017] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:54.899030] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193818808820, flush_io_finished_ret_:0, flush_io_finished_round_:1973, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:54.900840] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=106] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.900877] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.900901] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.900924] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:54.907488] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=265] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:54.909609] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [46305][T1_TimerWK1][T1][YB42C0A89F89-00062B2D1768577F-0-0] [lt=122] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=124) [2025-01-08 00:51:54.910055] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=103] get number(ret=0, service_type_=0, range=1, base_id=1736326314905288000, start_id=1736326314905288000, end_id=1736326314905288001) [2025-01-08 00:51:54.913519] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61E-0-0] [lt=29] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:54.913575] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61E-0-0] [lt=35] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326314664924000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326314664924000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:54.914897] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=96] ====== checkpoint timer task ====== [2025-01-08 00:51:54.915878] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:1, active_tx_count:1, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:54.915925] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=46] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1736326314905288000, v:0}) [2025-01-08 00:51:54.916136] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=25] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329254996000, latest_log_ts:{val:1736326312592647001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:54.916292] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=151] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329254996000, latest_log_ts:{val:1736326312592647001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:54.916352] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=53] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:54.916501] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=146] get rec log scn(service_type_=0, rec_log_ts={val:1736326312592647001, v:0}) [2025-01-08 00:51:54.916511] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:54.916518] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:54.916533] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:54.916553] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:54.916566] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:54.916755] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=178][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:54.916778] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:54.916789] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:54.929920] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84470022}], scn_array=[cnt:1, {val:1736326314905288000, v:0}], curr_size=545, accum_size=4518, time_cost=19376) [2025-01-08 00:51:54.944976] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:54.946855] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1736326314943418000, start_id=1736326314943418000, end_id=1736326314943418001) [2025-01-08 00:51:54.947596] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=146] Slow write(duration=10612, write_len=4096) [2025-01-08 00:51:54.947636] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=38] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8818, offset=2505152}, end_cursor:ObLogCursor{file_id=1, log_id=8819, offset=2505457}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cc4050, len:1600, log_data_len:305, seq:8818, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:54.952263] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E75-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:54.952346] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E75-0-0] [lt=61] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326314802629000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326314802629000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:54.953123] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=113] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326314836230000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=116884, in_cluster_service=true, cluster_version={val:1736326314606849000, v:0}, min_cluster_version={val:1736326314606849000, v:0}, max_cluster_version={val:1736326314606849000, v:0}, get_cluster_version_err=0, cluster_version_delta=346265, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=19111, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736326314422615000, v:0}, local_cluster_delta=530499, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:54.956961] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB61E-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1736326314954293000, start_id=1736326314954293000, end_id=1736326314954293001) [2025-01-08 00:51:54.957860] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48A7-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=25, avg_log_submit_to_flush_time=6305, avg_log_submit_to_slide_time=32003) [2025-01-08 00:51:54.958332] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:54.958382] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18378, miss_cnt:0, cache_read_size:2248300, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:54.960929] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=27] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12798291968, fast_delete_log_mode=false) [2025-01-08 00:51:54.965411] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:54.966902] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=36] [APPLY STAT CB APPEND COST TIME](cur_stat_count=62, stat_interval=5000000, avg (count/cost)=149, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:54.966928] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=25] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=62, stat_interval=5000000, avg (count/cost)=14593, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:54.966937] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=62, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:54.966944] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=62, stat_interval=5000000, avg (count/cost)=-1344, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:54.966951] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=62, stat_interval=5000000, avg (count/cost)=13398, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:54.968021] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:54.969161] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B49-0-0] [lt=24] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:54.969455] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B49-0-0] [lt=281] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1333, avg_group_log_size=121) [2025-01-08 00:51:54.969529] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B49-0-0] [lt=31] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:54.969565] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B49-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=62119, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:54.969654] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B49-0-0] [lt=51] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=16, avg_log_submit_to_flush_time=5676, avg_log_submit_to_slide_time=62103) [2025-01-08 00:51:54.969766] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=102] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:54.977418] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:54.977949] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43594274}], scn_array=[cnt:1, {val:1736326314965014000, v:0}], curr_size=1134, accum_size=4883, time_cost=11579) [2025-01-08 00:51:54.978855] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58611][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=182] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:51:54.979660] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEC-0-0] [lt=30] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:54.979710] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEC-0-0] [lt=40] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=52) [2025-01-08 00:51:54.984454] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=106] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:54.984614] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=21] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8819, offset=2505457}, end_cursor:ObLogCursor{file_id=1, log_id=8820, offset=2505754}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cc0050, len:1295, log_data_len:297, seq:8819, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:54.985488] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E75-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:54.985617] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E75-0-0] [lt=108] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326314802629000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326314802629000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:54.999244] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=38] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8820, offset=2505754}, end_cursor:ObLogCursor{file_id=1, log_id=8821, offset=2506051}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cd0050, len:998, log_data_len:297, seq:8820, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:55.001484] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=44] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8821, offset=2506051}, end_cursor:ObLogCursor{file_id=1, log_id=8822, offset=2506348}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cc8050, len:701, log_data_len:297, seq:8821, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:55.024596] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:219) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:55.024655] INFO [COMMON] handle (memory_dump.cpp:538) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=58] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:51:55.026009] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] update_check_range(min_check_version=377, max_check_version=378, global_version=379) [2025-01-08 00:51:55.033051] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=58] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5104) [2025-01-08 00:51:55.036164] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=178] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:55.040813] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58608][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=43] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-01-08 00:51:55.051614] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C8-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:55.051726] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C8-0-0] [lt=92] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326314647131003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326314647131003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:55.055803] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58640][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:55.055878] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58640][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48A9-0-0] [lt=41] init point thread id with(&point=0x5572d50e6d40, idx_=1296, point=[thread id=58640, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58640) [2025-01-08 00:51:55.056128] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:51:55.056735] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=64] PNIO [ratelimit] time: 1736326315056732, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008733, add_bytes: 0 [2025-01-08 00:51:55.065537] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:55.083366] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5310244378, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:55.083504] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=54] recycle one thread(this=0x2ad8885abcf0, total=4, remain=3) [2025-01-08 00:51:55.083520] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5310244378, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:55.085006] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=602][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43596746}], scn_array=[cnt:1, {val:1736326315050983000, v:0}], curr_size=588, accum_size=6809, time_cost=28335) [2025-01-08 00:51:55.085671] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=195] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:55.085930] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:55.086386] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2248422}], scn_array=[cnt:1, {val:1736326314647131004, v:0}], curr_size=122, accum_size=244, time_cost=30431) [2025-01-08 00:51:55.105363] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2236014}], scn_array=[cnt:1, {val:1736326314664924003, v:0}], curr_size=122, accum_size=244, time_cost=20988) [2025-01-08 00:51:55.105431] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2214542}], scn_array=[cnt:1, {val:1736326314802629002, v:0}], curr_size=122, accum_size=366, time_cost=24776) [2025-01-08 00:51:55.105509] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] succ to submit log task to replay service(to_submit_lsn={lsn:2248478}, to_submit_scn={val:1736326314647131004, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326315088832, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2248544}, next_to_submit_scn_:{val:1736326314647131005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2248422}, end_lsn:{lsn:2248544}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2248544}, readable_log_tail:{lsn:2248544}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64856224, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326314647131004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326314647131004, v:0}, data_checksum:3348837161, flag:20159}}, init_mode_version:0, accumulate_checksum:1820750981, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326314647131004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18380, miss_cnt:0, cache_read_size:2248544, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:55.106726] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18734245}], scn_array=[cnt:1, {val:1736326314802629002, v:0}], curr_size=122, accum_size=789, time_cost=23107) [2025-01-08 00:51:55.111122] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:55.111126] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58641][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:55.157039] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=41] ts response task statistics(alloc_count=60922, free_count=30460) [2025-01-08 00:51:55.157099] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=37] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326315157015], gts:1736326315152929000, latest_srr:[mts=1736326315157015]}) [2025-01-08 00:51:55.157121] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:55.157147] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326315157132], gts:1736326315152929000, latest_srr:[mts=1736326315157132]}) [2025-01-08 00:51:55.157529] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326315157510], gts:1736326315152929000, latest_srr:[mts=1736326315157510]}) [2025-01-08 00:51:55.157800] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:55.159455] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=77] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:55.162409] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=104] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:324K/76:309940 disk:324K/76:309940 t1m2:44K/10:372563 t1m2g0:44K/10:372563 t1002m2:176K/44:346667 t1002m2g0:176K/44:346667 t1001m2:104K/22:208023 t1001m2g0:104K/22:208023 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:55.165624] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=81][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:55.168488] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=53] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:55.172482] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2248544}], scn_array=[cnt:1, {val:1736326314647131005, v:0}], curr_size=122, accum_size=366, time_cost=10176) [2025-01-08 00:51:55.173495] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A1F-0-0] [lt=23] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=25, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:55.173521] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A1F-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=25625, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:55.175284] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:55.175408] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=91] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:55.175499] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=41] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:55.180556] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=71, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:55.180591] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:55.180656] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:55.180683] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:55.194517] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9658-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:55.204595] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2214664}], scn_array=[cnt:1, {val:1736326315152929000, v:0}], curr_size=122, accum_size=488, time_cost=23874) [2025-01-08 00:51:55.208344] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:55.209149] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=45] ====== check clog disk timer task ====== [2025-01-08 00:51:55.209239] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=84] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:55.209557] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18734367}], scn_array=[cnt:1, {val:1736326315152929000, v:0}], curr_size=122, accum_size=911, time_cost=13123) [2025-01-08 00:51:55.229659] INFO handle (memory_dump.cpp:665) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 154 261 0 1 2 68 1 0 1 4 1 1 0 1 5 43 5 0 1 7 1 2 0 1 8 48 1 0 1 12 7 3 0 1 13 1 1 0 1 14 1 2 0 1 15 1 1 0 1 16 1 1 0 1 18 4 3 0 1 20 1 2 0 500 0 94 238 0 500 7 7 4 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 2 1 0 500 17 9 7 0 500 22 79 61 0 500 23 23 10 0 508 0 2 10 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 100 230 0 1001 2 49 1 0 1001 4 1 1 0 1001 5 22 5 0 1001 8 30 1 0 1001 12 3 2 0 1001 13 1 1 0 1001 14 1 1 0 1001 18 4 3 0 1001 20 2 2 0 1002 0 163 254 0 1002 2 59 1 0 1002 4 2 1 0 1002 5 49 6 0 1002 7 11 2 0 1002 8 46 1 0 1002 12 2 3 0 1002 13 1 1 0 1002 14 1 1 0 1002 18 9 3 0 1002 20 1 2 0 cost_time: 203691 [2025-01-08 00:51:55.229725] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=42] [MEMORY] tenant: 1, limit: 2,147,483,648 hold: 1,049,079,808 cache_hold: 213,909,504 cache_used: 213,909,504 cache_item_count: 102 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 451,977,216 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 142,606,336 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 95,862,784 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 100,663,296 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 14,680,064 limit= 429,496,720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:55.230258] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 451,977,216 used= 347,836,208 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,211,520 used= 31,196,160 count= 15 avg_used= 2,079,744 block_cnt= 15 chunk_cnt= 15 mod=CACHE_MAP_NODE [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 30,760,960 used= 30,611,008 count= 26 avg_used= 1,177,346 block_cnt= 26 chunk_cnt= 20 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,196,912 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 37 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,268,096 used= 11,228,160 count= 4 avg_used= 2,807,040 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,618,560 used= 7,541,760 count= 10 avg_used= 754,176 block_cnt= 10 chunk_cnt= 8 mod=ResourceGroup [MEMORY] hold= 7,151,616 used= 6,144,000 count= 123 avg_used= 49,951 block_cnt= 123 chunk_cnt= 23 mod=[T]char [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMemCU [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,391,488 used= 2,971,968 count= 138 avg_used= 21,536 block_cnt= 138 chunk_cnt= 28 mod=DiagnosticInfo [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,780,352 used= 2,663,280 count= 586 avg_used= 4,544 block_cnt= 570 chunk_cnt= 29 mod=WorkerMap [MEMORY] hold= 2,768,896 used= 2,720,000 count= 13 avg_used= 209,230 block_cnt= 13 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,640 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 272 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,394,368 used= 1,372,072 count= 75 avg_used= 18,294 block_cnt= 75 chunk_cnt= 11 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,260,160 used= 1,199,264 count= 33 avg_used= 36,341 block_cnt= 33 chunk_cnt= 12 mod=CommSysVarFac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,179,648 used= 1,175,808 count= 8 avg_used= 146,976 block_cnt= 8 chunk_cnt= 6 mod=SQLSessionInfo [MEMORY] hold= 960,192 used= 936,448 count= 102 avg_used= 9,180 block_cnt= 102 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 933,888 used= 458,736 count= 57 avg_used= 8,048 block_cnt= 57 chunk_cnt= 18 mod=[T]ObPerfEventR [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 511,488 used= 490,752 count= 108 avg_used= 4,544 block_cnt= 108 chunk_cnt= 25 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 262,144 used= 150,272 count= 16 avg_used= 9,392 block_cnt= 16 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 262,144 used= 261,120 count= 4 avg_used= 65,280 block_cnt= 4 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 203,840 used= 194,432 count= 49 avg_used= 3,968 block_cnt= 49 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 13 mod=LinearHashMapNo [MEMORY] hold= 178,816 used= 174,592 count= 22 avg_used= 7,936 block_cnt= 22 chunk_cnt= 10 mod=SqlSession [MEMORY] hold= 155,024 used= 2,177 count= 1,920 avg_used= 1 block_cnt= 19 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 154,432 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 13 mod=QueryAllocator [MEMORY] hold= 147,456 used= 131,072 count= 2 avg_used= 65,536 block_cnt= 2 chunk_cnt= 2 mod=MinorMergeMgr [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 93,856 used= 24,336 count= 345 avg_used= 70 block_cnt= 79 chunk_cnt= 28 mod=Coro [MEMORY] hold= 89,408 used= 87,296 count= 11 avg_used= 7,936 block_cnt= 11 chunk_cnt= 8 mod=SqlSessiVarMap [MEMORY] hold= 89,408 used= 87,296 count= 11 avg_used= 7,936 block_cnt= 11 chunk_cnt= 6 mod=PlanVaIdx [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,216 used= 60,712 count= 4 avg_used= 15,178 block_cnt= 4 chunk_cnt= 4 mod=HB_SERVICE [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 5 mod=HashNodeLCSta [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 54,720 used= 51,840 count= 15 avg_used= 3,456 block_cnt= 12 chunk_cnt= 11 mod=CACHE_INST [MEMORY] hold= 52,736 used= 40,992 count= 61 avg_used= 672 block_cnt= 60 chunk_cnt= 22 mod=[T]ObWarningBuf [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 41,088 used= 28,672 count= 64 avg_used= 448 block_cnt= 32 chunk_cnt= 12 mod=OMT_Worker [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 38,240 used= 4,056 count= 169 avg_used= 24 block_cnt= 70 chunk_cnt= 28 mod=[T]MemoryContex [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PartCtx [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,640 used= 15,872 count= 4 avg_used= 3,968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnMemNode [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,936 used= 5,808 count= 11 avg_used= 528 block_cnt= 11 chunk_cnt= 8 mod=SqlSessiQuerSql [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 11 chunk_cnt= 5 mod=SequenceIdMap [MEMORY] hold= 7,488 used= 5,088 count= 12 avg_used= 424 block_cnt= 12 chunk_cnt= 6 mod=SequenceMap [MEMORY] hold= 6,880 used= 4,664 count= 11 avg_used= 424 block_cnt= 10 chunk_cnt= 6 mod=PackStateMap [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,864 used= 4,664 count= 11 avg_used= 424 block_cnt= 11 chunk_cnt= 7 mod=ContextsMap [MEMORY] hold= 6,848 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 5 mod=DICollector [MEMORY] hold= 4,864 used= 4,288 count= 3 avg_used= 1,429 block_cnt= 2 chunk_cnt= 2 mod=NegParamIdex [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,168 used= 768 count= 2 avg_used= 384 block_cnt= 2 chunk_cnt= 2 mod=TableLockNode [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 2 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 320 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSIDDLVar [MEMORY] hold= 347,836,208 used= 340,234,122 count= 31,603 avg_used= 10,765 mod=SUMMARY [2025-01-08 00:51:55.230322] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=49] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 142,606,336 used= 141,492,224 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 block_cnt= 68 chunk_cnt= 68 mod=Memstore [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:51:55.230342] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:51:55.230363] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 95,862,784 used= 90,371,200 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83,884,032 used= 75,811,934 count= 2,908 avg_used= 26,070 block_cnt= 2,908 chunk_cnt= 43 mod=SqlPhyPlan [MEMORY] hold= 4,709,056 used= 4,468,656 count= 313 avg_used= 14,276 block_cnt= 313 chunk_cnt= 41 mod=SqlPlanCache [MEMORY] hold= 1,353,472 used= 1,234,048 count= 622 avg_used= 1,984 block_cnt= 208 chunk_cnt= 39 mod=CreateContext [MEMORY] hold= 422,656 used= 412,672 count= 52 avg_used= 7,936 block_cnt= 52 chunk_cnt= 12 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 90,371,200 used= 81,928,910 count= 3,897 avg_used= 21,023 mod=SUMMARY [2025-01-08 00:51:55.230383] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,368 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,272 used= 58,751 count= 259 avg_used= 226 block_cnt= 21 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,368 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-01-08 00:51:55.230397] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 100,663,296 used= 98,058,240 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 98,058,240 used= 97,875,840 count= 190 avg_used= 515,136 block_cnt= 190 chunk_cnt= 48 mod=CoStack [MEMORY] hold= 98,058,240 used= 97,875,840 count= 190 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:51:55.230418] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 213,909,504 used= 213,909,504 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 mod=SUMMARY [2025-01-08 00:51:55.230436] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 14,680,064 used= 12,713,920 limit= 429,496,720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,036,352 used= 7,149,419 count= 109 avg_used= 65,591 block_cnt= 109 chunk_cnt= 5 mod=L_TabletPool [MEMORY] hold= 4,587,520 used= 4,569,600 count= 70 avg_used= 65,280 block_cnt= 70 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 12,713,920 used= 11,803,787 count= 181 avg_used= 65,214 mod=SUMMARY [2025-01-08 00:51:55.230450] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,584,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 block_cnt= 195 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:55.230465] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 16,256 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ExtInfoLog [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:55.230479] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 33,216 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 mod=SUMMARY [2025-01-08 00:51:55.230492] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 2,097,152 used= 49,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 49,152 used= 32,512 count= 2 avg_used= 16,256 block_cnt= 2 chunk_cnt= 1 mod=OB_LOG_PUSH_REQ [MEMORY] hold= 49,152 used= 32,512 count= 2 avg_used= 16,256 mod=SUMMARY [2025-01-08 00:51:55.230512] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 14,058,176 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 9,473,984 used= 9,445,184 count= 3 avg_used= 3,148,394 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 4,576,064 used= 4,467,968 count= 563 avg_used= 7,936 block_cnt= 563 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 14,058,176 used= 13,921,088 count= 567 avg_used= 24,552 mod=SUMMARY [2025-01-08 00:51:55.230530] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 73,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 73,152 used= 71,424 count= 9 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:51:55.230566] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 631,246,848 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 232,894,464 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 24,260,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 170,676,224 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98,557,952 limit= 9,223,372,036,854,775,807 [2025-01-08 00:51:55.231923] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 232,894,464 used= 186,015,728 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,397,376 used= 7,340,256 count= 7 avg_used= 1,048,608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7,259,504 used= 6,324,350 count= 1,032 avg_used= 6,128 block_cnt= 540 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,783,552 used= 5,752,960 count= 6 avg_used= 958,826 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 16 mod=Iteratorget_logical_del_time()=193820247301, cache_obj->added_lc()=false, cache_obj->get_object_id()=13690, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:55.340646] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] Garbage Collector is running(seq_=382, gc_interval=10000000) [2025-01-08 00:51:55.340833] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=120] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:55.340866] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=21] gc_check_member_list_ cost time(ret=0, time_us=99) [2025-01-08 00:51:55.340889] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=18] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:51:55.340904] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:55.340918] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:55.340928] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=12) [2025-01-08 00:51:55.340935] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] ObSafeDestroyHandler start process [2025-01-08 00:51:55.340949] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=12] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-01-08 00:51:55.340969] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-01-08 00:51:55.360969] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=389] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=216, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:55.361116] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=137] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:55.368221] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=91][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:55.372552] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18284, miss_cnt:0, cache_read_size:2236502, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:55.381910] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:55.382008] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=92] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:55.398814] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=73] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=90, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:55.408605] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AF3E-0-0] [lt=23] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.817513704299926758e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.517570905390110738e+02) [2025-01-08 00:51:55.427562] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1736326315152929000, v:0}, base_version={val:1736326315026454000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326315226587000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326315427539, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326315152929000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326315401341, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326315226587000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326315478046, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:55.430404] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4076, avg_group_log_size=370) [2025-01-08 00:51:55.437524] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58633][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A00-0-0] [lt=26] init point thread id with(&point=0x5572d50e6bc0, idx_=1290, point=[thread id=58633, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58633) [2025-01-08 00:51:55.459088] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:55.463021] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:55.468367] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=131] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:55.468401] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:55.478459] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:55.478565] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=95] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70914440, tenant_id=1001) [2025-01-08 00:51:55.478591] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:55.486249] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A2A-0-0] [lt=25] init point thread id with(&point=0x5572d50e6d80, idx_=1297, point=[thread id=58641, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58641) [2025-01-08 00:51:55.488948] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:55.489027] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:55.499833] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] ====== tenant freeze timer task ====== [2025-01-08 00:51:55.519937] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:55.541747] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:55.544320] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:55.557072] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48B3-0-0] [lt=34][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18477, log_task={header:{begin_lsn:{lsn:2248910}, end_lsn:{lsn:2249032}, log_id:18477, min_scn:{val:1736326315159571002, v:0}, max_scn:{val:1736326315159571002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2248788}, prev_proposal_id:1, committed_end_lsn:{lsn:2248910}, data_checksum:1715074492, accum_checksum:3429437161, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326315456018, freeze_ts:1736326315456018, submit_ts:1736326315456031, flushed_ts:1736326315464111, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:8080}, fs_cb_begin_ts=1736326315557034, log_life_time=101016) [2025-01-08 00:51:55.568724] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:55.633588] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:55.647419] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=129] current monitor number(seq_=-1) [2025-01-08 00:51:55.647751] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=77] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-01-08 00:51:55.648287] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=90] gc stale ls task succ [2025-01-08 00:51:55.648941] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1736326315648937, bytes: 884121895, bw: 0.241867 MB/s, add_ts: 1001543, add_bytes: 254007 [2025-01-08 00:51:55.651799] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:55.651834] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-01-08 00:51:55.651845] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:55.651859] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:55.651873] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193819565365, flush_io_finished_ret_:0, flush_io_finished_round_:1978, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:55.658763] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:55.661681] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:55.663771] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=212] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:55.663812] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:55.663832] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=16] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=110) [2025-01-08 00:51:55.668720] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:55.669504] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:55.674065] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D51-0-0] [lt=43] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:55.676164] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE948-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326315676044000, start_id=1736326315676044000, end_id=1736326315676044001) [2025-01-08 00:51:55.677578] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685517-0-0] [lt=11] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:51:55.677611] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685517-0-0] [lt=25] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:51:55.677638] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=50] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:55.678170] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:55.678294] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD9-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:55.678322] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CD9-0-0] [lt=26] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:55.680117] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=31] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:55.689508] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:55.689569] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=58] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:55.689891] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:55.690311] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58609][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=58] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:51:55.715522] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=96] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=43605956, this={dio_aligned_buf:{buf_write_offset:456, buf_padding_size:3640, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:46698, truncate_used_ts:13545}, log_block_size:67108864, io_fd:{first_id:-1, second_id:160, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=8192, aligned_block_offset=43601920, buf_len=516, total_write_size=96739, total_write_size_after_dio=757760, ob_pwrite_used_ts=926555, count=161, each_pwrite_cost=5755) [2025-01-08 00:51:55.740947] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=43] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:55.740972] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=24] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:55.741022] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84992-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:55.741041] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84992-0-0] [lt=16] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:55.746078] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC1-0-0] [lt=50][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:55.747686] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC1-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:55.758878] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=23] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:51:55.758935] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=52] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:55.758951] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:51:55.758962] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:55.760231] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=76] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:55.760279] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=35] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:55.760298] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=17] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:51:55.760309] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:51:55.760317] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:51:55.760324] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=98) [2025-01-08 00:51:55.761677] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:51:55.762528] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=847] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:55.769706] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:55.799050] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=80] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:55.809641] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=114] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=12440, avg_group_log_size=654) [2025-01-08 00:51:55.809691] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=12440) [2025-01-08 00:51:55.824801] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC428-0-0] [lt=71][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:55.827326] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=18] ====== check clog disk timer task ====== [2025-01-08 00:51:55.827356] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=24] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:55.829040] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:55.829083] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:55.830347] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=182] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=77, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:55.830438] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3963) [2025-01-08 00:51:55.835374] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:55.835427] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:51:55.835462] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:55.835576] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:51:55.835671] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1001}, {id:1002}]) [2025-01-08 00:51:55.839857] INFO pn_ratelimit (group.c:653) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=12] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:55.842110] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=85] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4602, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:55.842167] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=189, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:55.842255] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4548) [2025-01-08 00:51:55.842304] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=23, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:55.845579] INFO [LIB] stat (utility.h:1210) [58631][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D6D-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=25, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:55.845605] INFO [LIB] stat (utility.h:1210) [58631][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D6D-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7272, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:55.845616] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58631][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D6D-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=116, avg_log_submit_to_flush_time=4865, avg_log_submit_to_slide_time=7155) [2025-01-08 00:51:55.845712] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4776, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:55.851046] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=53687091, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=651907072, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=421834752, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:55.851283] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=104] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:55.851313] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=26] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2890, drift_size_=0, pre_drift_size=-2097152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:55.851421] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1116487680, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2104737792, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:55.851497] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=37] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:55.851527] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:55.860319] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=346] get number(ret=0, service_type_=0, range=1, base_id=1736326315858926000, start_id=1736326315858926000, end_id=1736326315858926001) [2025-01-08 00:51:55.869916] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:55.870667] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:55.870729] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-01-08 00:51:55.870747] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:55.870893] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:55.870925] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193819784343, flush_io_finished_ret_:0, flush_io_finished_round_:3802, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:55.873240] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:55.882854] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=2457, avg_group_log_size=223) [2025-01-08 00:51:55.882889] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2457) [2025-01-08 00:51:55.890012] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6726971392, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:55.890300] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:55.896739] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:55.896811] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:55.896856] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:55.897146] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:55.902400] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918544E-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:55.911126] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:55.911207] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:48303, [delay/us]:prepare:14, schedule:200, submit:111, rt:4026, total:4353) [2025-01-08 00:51:55.911303] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=89] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:55.911396] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=74] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48299, [delay/us]: prepare:14, schedule:200, submit:111, device:4026, total:4353) [2025-01-08 00:51:55.911425] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:55.911470] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:80501, [delay/us]:prepare:11, schedule:213, submit:215, rt:4128, total:4568) [2025-01-08 00:51:55.911481] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1001, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:55.911536] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:80510, [delay/us]: prepare:11, schedule:213, submit:215, device:4128, total:4568) [2025-01-08 00:51:55.911585] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:55.911618] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16101, [delay/us]:prepare:5, schedule:313, submit:1, rt:4044, total:4364) [2025-01-08 00:51:55.911645] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:3, hold_mem:24576 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:35, bw:156991, [delay/us]:prepare:11, schedule:275, submit:194, rt:5512, total:5994) [2025-01-08 00:51:55.911702] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=52] [IO STATUS TENANT](tenant_id=1002, ref_cnt=8, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49997, free_result_cnt=49997, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:55.911758] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=50] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16100, [delay/us]: prepare:5, schedule:313, submit:1, device:4044, total:4364) [2025-01-08 00:51:55.911779] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:36, avg_bw:156980, [delay/us]: prepare:11, schedule:275, submit:194, device:5512, total:5994) [2025-01-08 00:51:55.911817] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:55.911846] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4025, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:4571, total:4584) [2025-01-08 00:51:55.911877] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:55.911930] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4025, [delay/us]: prepare:0, schedule:0, submit:600000000, device:4571, total:4584) [2025-01-08 00:51:55.911950] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:55.911963] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=2, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 2 [2025-01-08 00:51:55.912008] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:55.912022] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:55.912033] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=218kB/s, net_out=218kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:55.970634] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=142][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:55.993497] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=198, gc_interval=10000000) [2025-01-08 00:51:55.993575] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=56] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:55.993616] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=28] gc_check_member_list_ cost time(ret=0, time_us=62) [2025-01-08 00:51:55.993637] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=15] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:51:55.996312] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:55.996361] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=43] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:55.996393] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=30] execute_gc cost time(ret=0, time_us=33) [2025-01-08 00:51:55.996406] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] ObSafeDestroyHandler start process [2025-01-08 00:51:56.008008] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] no need to update min/max version(new_version={val:1736326315152929004, v:0}, min_version={val:1736326315152929004, v:0}, max_version={val:1736326315152929004, v:0}) [2025-01-08 00:51:56.008106] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=72] get_max_applied_scn(scn={val:1736326315655664000, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2215396}, palf_committed_end_scn_:{val:1736326315655664000, v:0}, last_check_scn_:{val:1736326315708110000, v:0}, max_applied_cb_scn_:{val:1736326315655664000, v:0}}) [2025-01-08 00:51:56.008371] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=260] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2215396}, readable_log_tail:{lsn:2215396}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64889372, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:56.013024] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=26] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:56.025825] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-01-08 00:51:56.025922] WDIAG [RPC] wait (ob_async_rpc_proxy.h:453) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] execute rpc failed(rc=-4012, server="192.168.159.138:2882", timeout=200000, packet code=1861, arg={tenant_id:1002, tenant_info:{tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326313644270003, replayable_scn:1736326313644270003, readable_scn:1736326313644270003, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}, ora_rowscn:1736326314965014000, finish_data_version:0, data_version_barrier_scn:{val:0, v:0}}, ret=0) [2025-01-08 00:51:56.028088] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8717E-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:56.028148] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:56.028165] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:56.030911] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] no need to update min/max version(new_version={val:1736326315628725000, v:0}, min_version={val:1736326315628725000, v:0}, max_version={val:1736326315628725000, v:0}) [2025-01-08 00:51:56.036732] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:56.037393] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58628][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=36] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:51:56.075820] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=14] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:56.091071] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=102] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:56.091435] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:56.125191] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:51:56.125362] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=163] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:51:56.125378] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=12] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:51:56.125419] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=39] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:56.131115] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=36] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:51:56.133789] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=43] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:51:56.136035] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=41] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:51:56.136058] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=21] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:51:56.138463] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] no need to update min/max version(new_version={val:1736326315628725000, v:0}, min_version={val:1736326315628725000, v:0}, max_version={val:1736326315628725000, v:0}) [2025-01-08 00:51:56.141043] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084881-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:51:56.141142] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084881-0-0] [lt=38] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:56.141172] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084881-0-0] [lt=27] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:56.145761] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=17] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:51:56.148379] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=20] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:51:56.152763] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=20] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:51:56.152800] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=35] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:51:56.154953] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084881-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:51:56.155052] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084881-0-0] [lt=20] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:56.155139] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084881-0-0] [lt=85] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:56.159371] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=15] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:51:56.162415] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=22] get number(ret=0, service_type_=0, range=1, base_id=1736326316161938000, start_id=1736326316161938000, end_id=1736326316161938001) [2025-01-08 00:51:56.162536] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=37] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:56.166311] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=6] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:56.166435] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=483] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:56.167139] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=119] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:51:56.169517] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=34] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:51:56.169538] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=21] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:51:56.171801] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084881-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:51:56.171966] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084881-0-0] [lt=19] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:51:56.171986] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084881-0-0] [lt=19] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:51:56.172018] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=12] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:51:56.172830] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=809] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:56.172894] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=55] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:51:56.172909] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=14] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:56.172934] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084881-0-0] [lt=21] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:51:56.172963] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084881-0-0] [lt=28] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:51:56.172981] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=12] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:51:56.172991] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:56.173017] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084881-0-0] [lt=23] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:51:56.173046] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084881-0-0] [lt=28] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:51:56.173062] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=9] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:51:56.173072] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:51:56.173097] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084881-0-0] [lt=23] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:51:56.173117] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084881-0-0] [lt=19] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:51:56.175515] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:56.175547] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=29] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:56.175563] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=14] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:51:56.175598] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=11] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:51:56.175611] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:51:56.175620] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:51:56.175647] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=23] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:51:56.175667] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084881-0-0] [lt=19] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:51:56.177104] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=6239232) [2025-01-08 00:51:56.177965] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:56.180368] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=36] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:56.188306] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=55] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:56.204631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:56.149, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.238879] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] no need to update min/max version(new_version={val:1736326315628725000, v:0}, min_version={val:1736326315628725000, v:0}, max_version={val:1736326315628725000, v:0}) [2025-01-08 00:51:56.277385] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=47] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:51:56.277588] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50529][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=69][errcode=-4016] task executed too delay(delay_time=113800, *p_task_={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:0, time_interval:250.00ms, expected_run_time:00:51:56.408, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.278600] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=166] no need to update min/max version(new_version={val:1736326315152929004, v:0}, min_version={val:1736326315152929004, v:0}, max_version={val:1736326315152929004, v:0}) [2025-01-08 00:51:56.280559] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F89-00062B2D21F89E18-0-0] [lt=28][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5387, fly_ts=169399, send_timestamp=1736326316111156, peer="192.168.159.137:41270", sz=479) [2025-01-08 00:51:56.281135] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50527][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] task executed too delay(delay_time=176982, *p_task_={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:0, time_interval:100.00ms, expected_run_time:00:51:56.249, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.281856] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1736326316281853, bytes: 0, bw: 0.000000 MB/s, add_ts: 1225121, add_bytes: 0 [2025-01-08 00:51:56.283762] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:56.283819] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50522][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=338510, expected_run_ts_us_=1736326315945306, ObClockGenerator::getRealClock()=1736326316283817, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfcdb0, caller:mds_tenant_service.cpp:mtl_start:125, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8f0dfd090, time_wheel:0x2ad8dacdf990, thread_pool:0x2ad8dacdf170, is_running:1, time_interval:3.00s, expected_run_time:00:51:55.945, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.284551] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BBF-0-0] [lt=13] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:56.524095] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=121] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6714388480, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:56.524395] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:56.524503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] task scheduled too delay(scheduled_delay=418036, expected_run_ts_us_=1736326316106463, ObClockGenerator::getRealClock()=1736326316524501, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:56.106, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.524715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50380][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=0] task scheduled too delay(scheduled_delay=417040, expected_run_ts_us_=1736326316107663, ObClockGenerator::getRealClock()=1736326316524713, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:1, time_interval:100.00ms, expected_run_time:00:51:56.107, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.525304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=65][errcode=0] task scheduled too delay(scheduled_delay=275945, expected_run_ts_us_=1736326316249355, ObClockGenerator::getRealClock()=1736326316525302, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:56.249, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.525927] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.525977] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.525988] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=43] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.526146] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.526179] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=55] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.526547] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:56.528631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] task scheduled too delay(scheduled_delay=146102, expected_run_ts_us_=1736326316382524, ObClockGenerator::getRealClock()=1736326316528627, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8a0dff190, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff470, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:1, time_interval:500.00ms, expected_run_time:00:51:56.382, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.528750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=78][errcode=0] task scheduled too delay(scheduled_delay=120218, expected_run_ts_us_=1736326316408530, ObClockGenerator::getRealClock()=1736326316528749, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:56.408, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.529030] INFO [STORAGE] runTimerTask (ob_ddl_heart_beat_task.cpp:49) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=38] send to rs all task status succeed [2025-01-08 00:51:56.529192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] task scheduled too delay(scheduled_delay=254166, expected_run_ts_us_=1736326316275022, ObClockGenerator::getRealClock()=1736326316529190, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:56.275, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.530610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] task scheduled too delay(scheduled_delay=260466, expected_run_ts_us_=1736326316270140, ObClockGenerator::getRealClock()=1736326316530608, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8effff190, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff470, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:500.00ms, expected_run_time:00:51:56.270, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:56.531012] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46141][pnio1][T0][YB42C0A89F8A-00062B2D1769CA85-0-0] [lt=28][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1308, fly_ts=590308, send_timestamp=1736326315940701, peer="192.168.159.138:41998", sz=1588) [2025-01-08 00:51:56.533382] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8A-00062B2D1819C64B-0-0] [lt=19][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=330, fly_ts=592052, send_timestamp=1736326315941327, peer="192.168.159.138:42022", sz=251) [2025-01-08 00:51:56.533837] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3BCE-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=13029) [2025-01-08 00:51:56.535333] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [58625][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=96] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5312428638, tenant_id:500}, cur_thread_count=3, cnt=1, new_thread_count=4) [2025-01-08 00:51:56.535453] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=71007, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:56.535490] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2236990}], scn_array=[cnt:1, {val:1736326315665641001, v:0}], curr_size=122, accum_size=1220, time_cost=666227) [2025-01-08 00:51:56.535523] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:56.535548] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=540, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:56.535758] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT WRITE LOG COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=39263, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:56.535792] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43602965}], scn_array=[cnt:1, {val:1736326315858926000, v:0}], curr_size=589, accum_size=13029, time_cost=675211) [2025-01-08 00:51:56.535819] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=13029) [2025-01-08 00:51:56.535840] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=356, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:56.537120] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=120, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:56.537503] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=242] [PALF STAT FLUSH CB COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=72, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:56.537707] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=56] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=72, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:56.537776] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Slow write(duration=525223, write_len=4096) [2025-01-08 00:51:56.537795] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=17] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8822, offset=2506348}, end_cursor:ObLogCursor{file_id=1, log_id=8823, offset=2506653}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cd4050, len:404, log_data_len:305, seq:8822, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:56.538156] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84474798}], scn_array=[cnt:1, {val:1736326315849992000, v:0}], curr_size=513, accum_size=4158, time_cost=687451) [2025-01-08 00:51:56.538199] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4158) [2025-01-08 00:51:56.539544] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=72126, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:56.539567] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2215396}], scn_array=[cnt:1, {val:1736326315708110000, v:0}], curr_size=122, accum_size=1220, time_cost=656252) [2025-01-08 00:51:56.539586] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:56.539603] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=81, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:56.540578] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F5E2-0-0] [lt=8] get number(ret=0, service_type_=0, range=1, base_id=1736326316274758000, start_id=1736326316274758000, end_id=1736326316274758001) [2025-01-08 00:51:56.540643] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48B9-0-0] [lt=62] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:56.540686] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48B9-0-0] [lt=18] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:56.540704] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48B9-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:51:56.540729] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48B9-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=37, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:56.540738] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48B9-0-0] [lt=9] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=143901, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:56.540746] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48B9-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=9218, avg_log_submit_to_slide_time=102933) [2025-01-08 00:51:56.540756] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48B9-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18480, log_task={header:{begin_lsn:{lsn:2249276}, end_lsn:{lsn:2249398}, log_id:18480, min_scn:{val:1736326315646988001, v:0}, max_scn:{val:1736326315646988001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2249154}, prev_proposal_id:1, committed_end_lsn:{lsn:2249276}, data_checksum:1715074492, accum_checksum:373629691, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326315798807, freeze_ts:1736326315798808, submit_ts:1736326315798823, flushed_ts:1736326315804372, gen_to_freeze cost time:1, gen_to_submit cost time:16, submit_to_flush cost time:5549}, fs_cb_begin_ts=1736326316540719, log_life_time=741912) [2025-01-08 00:51:56.540787] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:56.540818] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18387, miss_cnt:0, cache_read_size:2249398, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:56.540827] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:56.540886] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=45] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=63, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:56.540905] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=18] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:56.541182] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=732) [2025-01-08 00:51:56.541289] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=102] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=32, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:56.541460] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:228K/52:134204 disk:228K/52:134204 t1m2:48K/11:112534 t1m2g0:48K/11:112534 t1002m2:132K/30:160628 t1002m2g0:132K/30:160628 t1001m2:48K/11:83810 t1001m2g0:48K/11:83810 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:56.541605] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] no need to update min/max version(new_version={val:1736326315152929004, v:0}, min_version={val:1736326315152929004, v:0}, max_version={val:1736326315152929004, v:0}) [2025-01-08 00:51:56.541735] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F8A-00062B2D1C49D0C1-0-0] [lt=21][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1308, fly_ts=597582, send_timestamp=1736326315944151, peer="192.168.159.138:42010", sz=1615) [2025-01-08 00:51:56.542827] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.542897] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E76-0-0] [lt=85] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:56.542921] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E76-0-0] [lt=15] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326315701616001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326315701616001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:56.544298] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=62] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.545677] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] no need to update min/max version(new_version={val:1736326315628725000, v:0}, min_version={val:1736326315628725000, v:0}, max_version={val:1736326315628725000, v:0}) [2025-01-08 00:51:56.546361] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4015] clock out of order(cur_ts=1736326316546358, cur_ts_=1736326316274758, delta=271600) [2025-01-08 00:51:56.554788] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6EB-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193821468613, cache_obj->added_lc()=false, cache_obj->get_object_id()=13691, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:56.556952] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43603554}], scn_array=[cnt:1, {val:1736326315858926001, v:0}], curr_size=122, accum_size=122, time_cost=17622) [2025-01-08 00:51:56.557604] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9335, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:56.557632] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2249398}], scn_array=[cnt:1, {val:1736326315646988002, v:0}], curr_size=122, accum_size=1220, time_cost=16850) [2025-01-08 00:51:56.557651] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:51:56.557682] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=436, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:56.558700] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=69, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:56.560068] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58643][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:56.560117] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:51:56.562007] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A966B-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=19, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:56.562026] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A966B-0-0] [lt=18] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=84268, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:56.562036] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A966B-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=18202, log_task={header:{begin_lsn:{lsn:2215396}, end_lsn:{lsn:2215518}, log_id:18202, min_scn:{val:1736326315708110000, v:0}, max_scn:{val:1736326315708110000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2215396}, data_checksum:2222604892, accum_checksum:2215046775, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326315883092, freeze_ts:1736326315883096, submit_ts:1736326315883251, flushed_ts:1736326316541174, gen_to_freeze cost time:4, gen_to_submit cost time:159, submit_to_flush cost time:657923}, fs_cb_begin_ts=1736326316561980, log_life_time=678888) [2025-01-08 00:51:56.563514] INFO [LIB] stat (utility.h:1210) [58629][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F38-0-0] [lt=49] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=14, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:56.563600] INFO [LIB] stat (utility.h:1210) [58629][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F38-0-0] [lt=84] [PALF STAT LOG LIFE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=45376, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:56.563613] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58629][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F38-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=20, avg_log_gen_to_freeze_time=948, avg_log_gen_to_submit_time=1065, avg_log_submit_to_flush_time=40133, avg_log_submit_to_slide_time=44310) [2025-01-08 00:51:56.563810] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58629][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F38-0-0] [lt=195][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=33261, log_task={header:{begin_lsn:{lsn:43602965}, end_lsn:{lsn:43603554}, log_id:33261, min_scn:{val:1736326315858926000, v:0}, max_scn:{val:1736326315858926000, v:0}, data_len:533, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43602965}, data_checksum:3032184051, accum_checksum:2825547970, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326315860389, freeze_ts:1736326315860404, submit_ts:1736326315860487, flushed_ts:1736326316537006, gen_to_freeze cost time:15, gen_to_submit cost time:98, submit_to_flush cost time:676519}, fs_cb_begin_ts=1736326316563503, log_life_time=703114) [2025-01-08 00:51:56.564114] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=157][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326315858926000, v:0}, log_ts:{val:1736326315858926000, v:0}, lsn:{lsn:43603021}, submit_ts:1736326315860384}, this:0x2ad8fb6fffd8, is_inited_:true, trans_id:{txid:67322}, ls_id:{id:1}, ctx:0x2ad8fb6fc5d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=703726, ctx_lock_wait_time=1) [2025-01-08 00:51:56.564412] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326315407928000, v:0}, new_max_version={val:1736326315407928000, v:0}, last_min_version={val:1736326315259360000, v:0}, last_max_version={val:1736326315259360000, v:0}, total_time_us=706050, delta_us=1156481) [2025-01-08 00:51:56.564453] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326315407928000, v:0}, min_version={val:1736326315407928000, v:0}, max_version={val:1736326315407928000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:56.566129] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBE13-0-0] [lt=32][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193821479962, cache_obj->added_lc()=false, cache_obj->get_object_id()=13692, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:56.567468] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:56.567755] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=13029, hit_cnt=20, read_cnt=20, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:56.568270] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58633][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D6E-0-0] [lt=48][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=45092, log_task={header:{begin_lsn:{lsn:84474798}, end_lsn:{lsn:84475311}, log_id:45092, min_scn:{val:1736326315849992000, v:0}, max_scn:{val:1736326315849992000, v:0}, data_len:457, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84474798}, data_checksum:3108352939, accum_checksum:3414158912, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326315850477, freeze_ts:1736326315850482, submit_ts:1736326315850543, flushed_ts:1736326316539383, gen_to_freeze cost time:5, gen_to_submit cost time:66, submit_to_flush cost time:688840}, fs_cb_begin_ts=1736326316568209, log_life_time=717732) [2025-01-08 00:51:56.568609] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A33-0-0] [lt=31] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1424) [2025-01-08 00:51:56.568892] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58644][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:56.569042] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:51:56.569077] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:56.569796] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58596][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=31] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:56.571281] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=26][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326315849992000, v:0}, log_ts:{val:1736326315849992000, v:0}, lsn:{lsn:84474854}, submit_ts:1736326315850472}, this:0x2ad89c064758, is_inited_:true, trans_id:{txid:28242}, ls_id:{id:1}, ctx:0x2ad89c060d50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=720804, ctx_lock_wait_time=0) [2025-01-08 00:51:56.572995] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [58625][IO_SYNC_CH2][T0][Y0-0000000000000000-0-0] [lt=76] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5312432292, tenant_id:500}, cur_thread_count=3, cnt=1, new_thread_count=4) [2025-01-08 00:51:56.577544] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=74330, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:56.577612] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=65][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18735913}], scn_array=[cnt:1, {val:1736326315708110000, v:0}], curr_size=122, accum_size=2457, time_cost=694553) [2025-01-08 00:51:56.577657] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2457) [2025-01-08 00:51:56.577695] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2091, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:56.577866] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:56.577890] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=79006, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:56.577901] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=14, avg_log_gen_to_submit_time=106, avg_log_submit_to_flush_time=76641, avg_log_submit_to_slide_time=78900) [2025-01-08 00:51:56.577931] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=19842, log_task={header:{begin_lsn:{lsn:18735913}, end_lsn:{lsn:18736035}, log_id:19842, min_scn:{val:1736326315708110000, v:0}, max_scn:{val:1736326315708110000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18735913}, data_checksum:667309872, accum_checksum:1993373859, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326315882768, freeze_ts:1736326315882772, submit_ts:1736326315882820, flushed_ts:1736326316577760, gen_to_freeze cost time:4, gen_to_submit cost time:52, submit_to_flush cost time:694940}, fs_cb_begin_ts=1736326316577783, log_life_time=695015) [2025-01-08 00:51:56.577985] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=42, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:56.578327] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1912, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:56.579329] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F39-0-0] [lt=17][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=33262, log_task={header:{begin_lsn:{lsn:43603554}, end_lsn:{lsn:43603676}, log_id:33262, min_scn:{val:1736326315858926001, v:0}, max_scn:{val:1736326315858926001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43602965}, data_checksum:667309872, accum_checksum:1561175739, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326316003835, freeze_ts:1736326316537025, submit_ts:1736326316537365, flushed_ts:1736326316557265, gen_to_freeze cost time:533190, gen_to_submit cost time:533530, submit_to_flush cost time:19900}, fs_cb_begin_ts=1736326316579305, log_life_time=575470) [2025-01-08 00:51:56.586384] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0C2-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:56.625294] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=8, avg_log_batch_cnt=1, total_group_log_size=2986, avg_group_log_size=373) [2025-01-08 00:51:56.625673] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:56.626669] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BC1-0-0] [lt=29] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:56.627035] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [50463][T1002_ArcFetche][T1002][YB42C0A89F89-00062B2D1C18458E-0-0] [lt=27] ObArchiveFetcher is running(thread_index=0) [2025-01-08 00:51:56.628833] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1003, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18379, max_lsn:2237112, max_scn:{val:1736326315665641001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2237112}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2236990}, last_submit_end_lsn:{lsn:2237112}, last_submit_log_id:18379, last_submit_log_pid:1, max_flushed_lsn:{lsn:2236990}, max_flushed_end_lsn:{lsn:2237112}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2236990}, last_slide_log_id:18378, last_slide_scn:{val:1736326315665641000, v:0}, last_slide_lsn:{lsn:2236868}, last_slide_end_lsn:{lsn:2236990}, last_slide_log_pid:1, last_slide_log_accum_checksum:1723290481, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924fd6530}, [StateMgr]={this:0x2ad924fda830, self:"192.168.159.137:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.139:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18379, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad924fe12f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326314171314, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326314171314, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924fd7c70}, [ModeMgr]={palf_id:1003, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436825152001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/meta", dir_fd:181, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924fdca38, flashback_version:0}, log_storage:{log_tail:{lsn:2237112}, readable_log_tail:{lsn:2237112}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64867656, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, palf_epoch:4, last_purge_throttling_ts:-1, this:0x2ad924fdaf70}, [Reconfirm]={palf_id:1003, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924fda9b0}) [2025-01-08 00:51:56.629238] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=342][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:56.629259] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:51:56.629271] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:51:56.629281] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] current log_task status(palf_id=1003, self="192.168.159.137:2882", log_id=18379, log_task={header:{begin_lsn:{lsn:2236990}, end_lsn:{lsn:2237112}, log_id:18379, min_scn:{val:1736326315665641001, v:0}, max_scn:{val:1736326315665641001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2236868}, prev_proposal_id:1, committed_end_lsn:{lsn:2236990}, data_checksum:940009244, accum_checksum:2373523856, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1736326315869138, freeze_ts:1736326315869138, submit_ts:1736326315869145, flushed_ts:1736326316537673, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:668528}, ack_info_list=[]) [2025-01-08 00:51:56.630125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5990) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=39] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000) [2025-01-08 00:51:56.630153] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.630305] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.630433] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.630512] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.630572] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.630682] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.630769] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.630834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.630928] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.631022] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.631115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.631301] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:101011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.631388] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.631532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.631618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.631705] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.631896] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.631969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.632096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.632229] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101098}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.632448] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=75] get multi version start(ls_id={id:1}, tablet_id={id:419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.632573] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.632900] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.632979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633139] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:101036}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633381] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=118] get multi version start(ls_id={id:1}, tablet_id={id:60005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633503] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633570] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101064}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633659] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101080}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633783] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.633959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101083}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634189] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634335] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634454] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101018}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634512] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634642] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634835] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101034}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.634952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101084}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635182] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635302] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635547] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635604] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101068}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635809] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=88] get multi version start(ls_id={id:1}, tablet_id={id:468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.635956] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=66] get multi version start(ls_id={id:1}, tablet_id={id:157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636302] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636656] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636722] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101073}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636894] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.636952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101099}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101092}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637067] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637301] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637423] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637486] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637591] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101031}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.637770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=85] get multi version start(ls_id={id:1}, tablet_id={id:60425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638158] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=108] get multi version start(ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638391] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638511] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638570] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638689] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=65] get multi version start(ls_id={id:1}, tablet_id={id:50325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.638945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639023] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101067}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639200] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=106] get multi version start(ls_id={id:1}, tablet_id={id:60422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101027}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639769] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639827] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.639954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.640040] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:60355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.640101] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.640159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.644989] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B5D-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:56.645027] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B5D-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:56.645044] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B5D-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:51:56.645070] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B5D-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:56.645079] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B5D-0-0] [lt=9] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=92998, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:56.645087] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B5D-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=45, avg_log_submit_to_flush_time=71852, avg_log_submit_to_slide_time=92953) [2025-01-08 00:51:56.645098] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B5D-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18379, log_task={header:{begin_lsn:{lsn:2236990}, end_lsn:{lsn:2237112}, log_id:18379, min_scn:{val:1736326315665641001, v:0}, max_scn:{val:1736326315665641001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2236868}, prev_proposal_id:1, committed_end_lsn:{lsn:2236990}, data_checksum:940009244, accum_checksum:2373523856, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326315869138, freeze_ts:1736326315869138, submit_ts:1736326315869145, flushed_ts:1736326316537673, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:668528}, fs_cb_begin_ts=1736326316645060, log_life_time=775922) [2025-01-08 00:51:56.645328] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:56.645366] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18289, miss_cnt:0, cache_read_size:2237112, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:56.645437] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=179] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=61, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:56.645485] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:56.645543] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=43, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:56.645560] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=5, avg_log_batch_cnt=1, total_group_log_size=610, avg_group_log_size=122) [2025-01-08 00:51:56.645575] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=610) [2025-01-08 00:51:56.645587] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:56.645646] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:56.645982] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=217][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84475433}], scn_array=[cnt:1, {val:1736326316161938000, v:0}], curr_size=122, accum_size=244, time_cost=20060) [2025-01-08 00:51:56.647093] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=6784] get multi version start(ls_id={id:1}, tablet_id={id:101082}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647174] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:50263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647308] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:50115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647430] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647489] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647753] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=71] get multi version start(ls_id={id:1}, tablet_id={id:50420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647868] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647889] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AF47-0-0] [lt=37] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.745514345169067383e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=387, Throttle Trigger(MB)=232, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.799183097836414618e+02) [2025-01-08 00:51:56.647935] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.647993] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648051] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101057}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648298] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] no need to update min/max version(new_version={val:1736326315628725000, v:0}, min_version={val:1736326315628725000, v:0}, max_version={val:1736326315628725000, v:0}) [2025-01-08 00:51:56.648333] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:56.648339] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648521] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648579] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648800] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101033}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648868] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.648984] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=94] get multi version start(ls_id={id:1}, tablet_id={id:60181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649191] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101066}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649316] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:50305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649376] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649496] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=65] get multi version start(ls_id={id:1}, tablet_id={id:232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649729] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.649932] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101030}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.650002] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:101020}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.650060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.650229] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101044}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.650296] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:100002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.650415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.650662] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.650851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=72] get multi version start(ls_id={id:1}, tablet_id={id:50440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.651041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.651292] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=80] get multi version start(ls_id={id:1}, tablet_id={id:293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.651437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.651592] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.651713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.651777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.651835] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101013}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652258] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=55] get multi version start(ls_id={id:1}, tablet_id={id:192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652504] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652688] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101072}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652747] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101088}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652879] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.652938] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101069}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653129] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653239] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653302] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653418] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101060}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=68] get multi version start(ls_id={id:1}, tablet_id={id:252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101100}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653828] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101022}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.653944] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654070] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=71] get multi version start(ls_id={id:1}, tablet_id={id:50110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654129] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654246] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654336] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654399] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654457] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654751] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:101104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.654994] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655181] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101035}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655362] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=92] get multi version start(ls_id={id:1}, tablet_id={id:5}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:60327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655644] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101058}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.655913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:101081}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656180] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=30] get multi version start(ls_id={id:1}, tablet_id={id:101086}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656391] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101052}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656453] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656514] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656574] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101019}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101079}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656912] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.656995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.657059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.657118] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.657886] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1736326316657882, bytes: 884209248, bw: 0.082568 MB/s, add_ts: 1008945, add_bytes: 87353 [2025-01-08 00:51:56.657993] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=29] gc stale ls task succ [2025-01-08 00:51:56.658093] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43603676}], scn_array=[cnt:1, {val:1736326316161938000, v:0}], curr_size=122, accum_size=244, time_cost=31717) [2025-01-08 00:51:56.659741] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2215518}], scn_array=[cnt:1, {val:1736326316161938000, v:0}], curr_size=122, accum_size=122, time_cost=11251) [2025-01-08 00:51:56.661698] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:56.661731] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-01-08 00:51:56.661741] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:56.661756] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:56.661770] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193820565746, flush_io_finished_ret_:0, flush_io_finished_round_:1979, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:56.663630] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=520, get_gts_with_stc_cnt=78, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=53, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:56.663663] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=462, get_gts_with_stc_cnt=70, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=66, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:56.657177] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.664946] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=7652] get multi version start(ls_id={id:1}, tablet_id={id:406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:1}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665342] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665465] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665533] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.665959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666293] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:50438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666356] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666468] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666689] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666750] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666811] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666933] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.666994] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667055] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667256] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667436] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667497] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667768] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667827] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.667945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668178] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668310] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668377] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668573] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:60150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.668942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101043}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.669000] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.669059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.669118] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.669176] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.669364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.669520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=61] get multi version start(ls_id={id:1}, tablet_id={id:50482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.669806] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:60151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.670020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=82] get multi version start(ls_id={id:1}, tablet_id={id:50232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.670169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:50236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.670239] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.670303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.670602] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=129] get multi version start(ls_id={id:1}, tablet_id={id:189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.670741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.671248] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=328] get multi version start(ls_id={id:1}, tablet_id={id:101032}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.671378] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.671508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.671633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.671816] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.671912] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=30] get multi version start(ls_id={id:1}, tablet_id={id:528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.671988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:50331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.672115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=73] get multi version start(ls_id={id:1}, tablet_id={id:465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.672252] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:50332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.672324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.672390] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.672568] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=71] get multi version start(ls_id={id:1}, tablet_id={id:107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.672632] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.672690] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.672913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.672914] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:56.672982] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673283] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101028}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673344] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673459] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673516] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673575] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101021}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673689] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673746] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673860] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.673975] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.674031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.674088] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.674144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.674201] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.685453] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDA-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:56.685545] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDA-0-0] [lt=88] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:56.686364] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58633][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD94-0-0] [lt=39] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326316752072, keepalive_ts:-1}) [2025-01-08 00:51:56.686397] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58633][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD94-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326316752072, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:56.688412] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58640][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DAD3-0-0] [lt=22][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18481, log_task={header:{begin_lsn:{lsn:2249398}, end_lsn:{lsn:2249520}, log_id:18481, min_scn:{val:1736326315646988002, v:0}, max_scn:{val:1736326315646988002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2249276}, prev_proposal_id:1, committed_end_lsn:{lsn:2249398}, data_checksum:1715074492, accum_checksum:3572130189, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326316540637, freeze_ts:1736326316540637, submit_ts:1736326316540663, flushed_ts:1736326316558658, gen_to_freeze cost time:0, gen_to_submit cost time:26, submit_to_flush cost time:17995}, fs_cb_begin_ts=1736326316688401, log_life_time=147764) [2025-01-08 00:51:56.688952] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58644][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3BD2-0-0] [lt=17] init point thread id with(&point=0x5572d50e6e40, idx_=1300, point=[thread id=58644, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58644) [2025-01-08 00:51:56.692018] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:56.692188] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=41] [APPLY STAT CB APPEND COST TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=154, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:51:56.692250] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=61] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=15115, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:51:56.692258] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:51:56.692265] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=-1374, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:51:56.692272] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=13895, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:51:56.696014] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:56.696118] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=58][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43603798}], scn_array=[cnt:1, {val:1736326316645863000, v:0}], curr_size=589, accum_size=833, time_cost=36594) [2025-01-08 00:51:56.696356] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:56.696866] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:56.697124] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:56.698957] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.702899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:60522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.702976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.703364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=246] get multi version start(ls_id={id:1}, tablet_id={id:50528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.703528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:60528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.703628] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.704077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=312] get multi version start(ls_id={id:1}, tablet_id={id:50375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.704238] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101037}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.704355] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=57] get multi version start(ls_id={id:1}, tablet_id={id:263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.704454] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.704622] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.704739] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:101054}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.704979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=64] get multi version start(ls_id={id:1}, tablet_id={id:153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.705298] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.707509] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B4BBAE8-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326316704896000, start_id=1736326316704896000, end_id=1736326316704896001) [2025-01-08 00:51:56.715394] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:60433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.715530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=75] get multi version start(ls_id={id:1}, tablet_id={id:50005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.715717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101041}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.715941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=55] get multi version start(ls_id={id:1}, tablet_id={id:50369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.716041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101097}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.716472] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.717077] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43604387}], scn_array=[cnt:1, {val:1736326316664324000, v:0}], curr_size=1666, accum_size=2499, time_cost=20474) [2025-01-08 00:51:56.718431] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61F-0-0] [lt=19] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:56.718467] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB61F-0-0] [lt=23] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326316274758001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326316274758001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:56.721328] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46386][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BED7-0-0] [lt=42] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:51:56.724871] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18736157}], scn_array=[cnt:1, {val:1736326316704896000, v:0}], curr_size=545, accum_size=667, time_cost=17145) [2025-01-08 00:51:56.727060] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=57] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6722756608, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:56.728142] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:56.717781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=1200] get multi version start(ls_id={id:1}, tablet_id={id:101024}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.730656] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=12751] get multi version start(ls_id={id:1}, tablet_id={id:50147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.730801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.730875] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.730939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.731017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.731101] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.731404] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=73] get multi version start(ls_id={id:1}, tablet_id={id:50190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.731470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.731532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.731690] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101055}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.731896] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.731969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.732097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.732243] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.732350] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.732498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=88] get multi version start(ls_id={id:1}, tablet_id={id:50433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.732562] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.732645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.732713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.733088] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1AD-0-0] [lt=16] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:51:56.733120] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1AD-0-0] [lt=25] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:56.733147] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58398][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1AD-0-0] [lt=24] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:51:56.733669] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101063}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.733775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.733982] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.734048] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:3}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.734105] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.734365] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.734435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.734560] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101040}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.734913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.734990] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.737380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.737567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.737647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.737841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101047}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.738156] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.738327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.738397] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.738623] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=111] get multi version start(ls_id={id:1}, tablet_id={id:60505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.738687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.738747] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.738807] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101059}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.738866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.738954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:50262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.739016] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.739149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.739275] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.739340] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.755479] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:56.762838] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:195) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC2-0-0] [lt=30][errcode=-4018] table_operator_ get_source failed(ret=-4018) [2025-01-08 00:51:56.764055] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC2-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:56.765596] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC2-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:56.765731] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326315849992000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=915731, in_cluster_service=true, cluster_version={val:1736326315628725000, v:0}, min_cluster_version={val:1736326315628725000, v:0}, max_cluster_version={val:1736326315628725000, v:0}, get_cluster_version_err=0, cluster_version_delta=1136998, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36864, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=571262710, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:56.767418] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=117] ====== tenant freeze timer task ====== [2025-01-08 00:51:56.777595] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A09-0-0] [lt=24] init point thread id with(&point=0x5572d50e6e00, idx_=1299, point=[thread id=58643, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58643) [2025-01-08 00:51:56.777947] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4015] clock out of order(cur_ts=1736326316777945, cur_ts_=1736326316738491, delta=39454) [2025-01-08 00:51:56.782440] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=17] Slow write(duration=53389, write_len=8192) [2025-01-08 00:51:56.782460] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=16] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8823, offset=2506653}, end_cursor:ObLogCursor{file_id=1, log_id=8824, offset=2506950}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ccc050, len:4195, log_data_len:297, seq:8823, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:56.783075] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E76-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:56.783109] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E76-0-0] [lt=19] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326316161938000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326316161938000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:56.783789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=151] get multi version start(ls_id={id:1}, tablet_id={id:262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784093] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784254] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784320] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784494] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784724] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.784952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785066] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785301] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=81] get multi version start(ls_id={id:1}, tablet_id={id:60143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785418] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101046}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:50464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785833] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.785948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786078] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786141] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786199] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786320] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786490] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:60191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786607] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786665] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101045}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786722] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786779] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786865] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786930] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.786989] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787105] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787162] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787283] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:60182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787440] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787507] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=77] get multi version start(ls_id={id:1}, tablet_id={id:101078}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787849] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=66] get multi version start(ls_id={id:1}, tablet_id={id:148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.787909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.788020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.788080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101049}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.788268] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.788387] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A6-0-0] [lt=23] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12274, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:56.788495] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.788627] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.788845] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=161] get multi version start(ls_id={id:1}, tablet_id={id:50209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.789050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101101}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.789316] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=90] get multi version start(ls_id={id:1}, tablet_id={id:221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.789509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=53] get multi version start(ls_id={id:1}, tablet_id={id:50255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.789706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:50522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.789785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.789970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.790143] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.790379] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.790501] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.790569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.790753] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=73] get multi version start(ls_id={id:1}, tablet_id={id:101091}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.790827] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.790942] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=59] get multi version start(ls_id={id:1}, tablet_id={id:60220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791066] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101039}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791245] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101029}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791315] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791377] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791559] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791746] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791807] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791867] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.791928] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792053] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:60261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792168] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=84] get multi version start(ls_id={id:1}, tablet_id={id:333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792569] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792574] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A6-0-0] [lt=38] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12274, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326316792550, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326316792550, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326316792550, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:56.792631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792619] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A6-0-0] [lt=42] success to process renew task(task={tenant_id:1, table_id:12274, add_timestamp:1736326316786106}, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326316792550, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326316792550, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326316792550, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:56.792718] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:100004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.792935] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:60250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.793005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.793096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.793174] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101095}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.795307] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2249520}], scn_array=[cnt:1, {val:1736326316274758000, v:0}], curr_size=122, accum_size=122, time_cost=62246) [2025-01-08 00:51:56.795386] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84475555}], scn_array=[cnt:1, {val:1736326316657283000, v:0}], curr_size=122, accum_size=366, time_cost=66534) [2025-01-08 00:51:56.799059] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101070}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.799152] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.799865] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.799948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.800031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101075}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.800051] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E76-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326316782750000, start_id=1736326316782750000, end_id=1736326316782750001) [2025-01-08 00:51:56.800104] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.800166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.800620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=394] get multi version start(ls_id={id:1}, tablet_id={id:60197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.800741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.800804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101077}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.800864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.800924] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.800983] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101071}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801075] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801139] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801262] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801330] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:101065}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801680] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801740] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801798] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.801982] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=126] get multi version start(ls_id={id:1}, tablet_id={id:329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802105] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:100003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802167] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802434] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802492] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101014}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802550] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802607] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802665] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802883] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.802987] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803048] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803106] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803289] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803351] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803547] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803748] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:50221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803819] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.803878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804026] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804517] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804580] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804669] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804763] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:101050}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.804992] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=146] get multi version start(ls_id={id:1}, tablet_id={id:50407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.805242] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=89] get multi version start(ls_id={id:1}, tablet_id={id:100001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.805490] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=105] get multi version start(ls_id={id:1}, tablet_id={id:50284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.805737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=137] get multi version start(ls_id={id:1}, tablet_id={id:243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.805830] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.805892] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.805952] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.806065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.806336] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101056}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.806464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.806594] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.806707] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.806771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.806855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.806949] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807011] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101023}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807184] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101048}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=90] get multi version start(ls_id={id:1}, tablet_id={id:330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:101026}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807479] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807542] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:50215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807892] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.807951] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808128] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808265] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=56] get multi version start(ls_id={id:1}, tablet_id={id:101025}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808467] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808531] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:4}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808866] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:100005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.808927] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.809015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.809132] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.809201] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.809358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=99] get multi version start(ls_id={id:1}, tablet_id={id:60377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.809421] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.809500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.809563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.809618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.810743] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2215640}], scn_array=[cnt:1, {val:1736326316704896000, v:0}], curr_size=122, accum_size=244, time_cost=43249) [2025-01-08 00:51:56.812463] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [50625][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D0C3-0-0] [lt=91] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326316812263, can_tenant_elr:true}) [2025-01-08 00:51:56.813756] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB APPEND COST TIME](cur_stat_count=12, stat_interval=5000000, avg (count/cost)=94, this=0x2ad934926280, extra_info="") [2025-01-08 00:51:56.813804] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=48] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=12, stat_interval=5000000, avg (count/cost)=70579, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:51:56.813817] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=12, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:51:56.813824] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=12, stat_interval=5000000, avg (count/cost)=-3562, this=0x2ad934926910, extra_info="") [2025-01-08 00:51:56.813831] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=12, stat_interval=5000000, avg (count/cost)=67111, this=0x2ad934926b40, extra_info="") [2025-01-08 00:51:56.809716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=4337] get multi version start(ls_id={id:1}, tablet_id={id:60242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814462] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101085}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814619] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814774] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814940] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.814999] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815114] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815368] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:101087}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815431] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815491] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815671] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815731] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815791] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.815887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.817926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=1968] get multi version start(ls_id={id:1}, tablet_id={id:60232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:101051}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818162] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:100006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818277] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=59] get multi version start(ls_id={id:1}, tablet_id={id:60382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818429] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101061}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818691] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818755] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818849] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.818947] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101076}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819102] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819188] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819336] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=90] get multi version start(ls_id={id:1}, tablet_id={id:50107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819488] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819543] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819632] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819725] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.819818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.820084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=94] get multi version start(ls_id={id:1}, tablet_id={id:425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.820503] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=55] get multi version start(ls_id={id:1}, tablet_id={id:421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.820790] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=212] get multi version start(ls_id={id:1}, tablet_id={id:60221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.820984] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.821279] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101074}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.821402] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.821636] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:60192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.821743] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:50323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.821808] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101053}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.821901] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.821966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.822053] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.822258] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:50105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.822475] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:50237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.822635] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.822710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.822773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.822965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=81] get multi version start(ls_id={id:1}, tablet_id={id:50352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.823087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.823031] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43606053}], scn_array=[cnt:1, {val:1736326316782750000, v:0}], curr_size=750, accum_size=3249, time_cost=21674) [2025-01-08 00:51:56.823707] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.823817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:50112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.823886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.823949] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824085] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824241] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824331] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824477] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824540] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824600] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824660] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824840] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824900] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.824959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825152] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:60169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825281] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101042}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825400] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0C5-0-0] [lt=70][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:56.825460] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825709] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825769] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825828] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101096}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825888] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.825971] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826034] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826094] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826477] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826759] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101038}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.826950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.827094] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.828475] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2237234}], scn_array=[cnt:1, {val:1736326316274758002, v:0}], curr_size=122, accum_size=244, time_cost=46454) [2025-01-08 00:51:56.833959] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=18] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8824, offset=2506950}, end_cursor:ObLogCursor{file_id=1, log_id=8825, offset=2507247}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cd8050, len:3898, log_data_len:297, seq:8824, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:56.836536] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3782) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-01-08 00:51:56.836571] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3805) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=33] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-01-08 00:51:56.836894] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=12] gc stale ls task succ [2025-01-08 00:51:56.837056] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1003}, {id:1002}]) [2025-01-08 00:51:56.838466] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=88] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8825, offset=2507247}, end_cursor:ObLogCursor{file_id=1, log_id=8826, offset=2507544}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cdc050, len:3601, log_data_len:297, seq:8825, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:56.841992] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:56.842047] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:56.842389] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT WRITE LOG COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=165221, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:56.842413] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84475677}], scn_array=[cnt:1, {val:1736326316738491000, v:0}], curr_size=586, accum_size=952, time_cost=45475) [2025-01-08 00:51:56.842443] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=513, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:56.842599] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:56.842647] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:51:56.842672] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:56.842708] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=37] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:56.842728] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=192) [2025-01-08 00:51:56.842778] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:51:56.842804] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=19] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:51:56.842819] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=194) [2025-01-08 00:51:56.842858] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2279) [2025-01-08 00:51:56.842911] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:56.844221] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:60260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.844489] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.844576] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.844644] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101090}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.844728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.844845] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.844913] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.845030] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=59] get multi version start(ls_id={id:1}, tablet_id={id:373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324505683208000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324505683208000, max_merged_snapshot=1) [2025-01-08 00:51:56.847049] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:116) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685518-0-0] [lt=16] SSTableGCTask(cost_ts=217503) [2025-01-08 00:51:56.855991] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:56.856814] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87184-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:56.856873] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:56.856885] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:56.858183] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=7, avg_log_batch_cnt=1, total_group_log_size=4427, avg_group_log_size=632) [2025-01-08 00:51:56.858292] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=103] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4427) [2025-01-08 00:51:56.862871] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=22] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:56.864813] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=50] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:56.864934] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=47] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326314412287000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:56.867735] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC42B-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:56.876770] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:56.878179] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=52] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:56.878346] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=89] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326315701616000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:56.878374] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326315701616000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326315701616000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:51:56.879585] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C9-0-0] [lt=20] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:56.879621] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2C9-0-0] [lt=19] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326316274758001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326316274758001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:56.881434] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=88] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:56.884165] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=37] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:56.895085] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B61-0-0] [lt=54][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18381, log_task={header:{begin_lsn:{lsn:2237234}, end_lsn:{lsn:2237356}, log_id:18381, min_scn:{val:1736326316274758002, v:0}, max_scn:{val:1736326316274758002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2237112}, prev_proposal_id:1, committed_end_lsn:{lsn:2237234}, data_checksum:940009244, accum_checksum:3421033375, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326316781960, freeze_ts:1736326316781960, submit_ts:1736326316781974, flushed_ts:1736326316829369, gen_to_freeze cost time:0, gen_to_submit cost time:14, submit_to_flush cost time:47395}, fs_cb_begin_ts=1736326316895072, log_life_time=113112) [2025-01-08 00:51:56.896534] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=47] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.896573] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.896598] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.897326] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=229] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:56.904802] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918544F-0-0] [lt=30][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:56.905062] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:56.905085] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-01-08 00:51:56.905095] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:56.905109] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:56.905122] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193820816655, flush_io_finished_ret_:0, flush_io_finished_round_:1975, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:56.905756] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=21] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:51:56.905780] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=6) [2025-01-08 00:51:56.905790] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=9] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:51:56.905799] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=6] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:51:56.927556] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6720679936, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:56.928089] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=11] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:56.928291] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:56.929620] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=50] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:56.929852] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=49] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326315839366000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:56.930114] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5313352994, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:56.930262] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:56.930275] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8db9c33e0) [2025-01-08 00:51:56.930284] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5313352994, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:56.939048] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=36] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:51:56.941660] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16E9CA76-0-0] [lt=95] get number(ret=0, service_type_=0, range=1, base_id=1736326316939590000, start_id=1736326316939590000, end_id=1736326316939590001) [2025-01-08 00:51:56.941792] INFO [LIB] stat (utility.h:1210) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16E9CA76-0-0] [lt=29] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=50, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:56.941844] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16E9CA76-0-0] [lt=25] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2864) [2025-01-08 00:51:56.941862] INFO [LIB] stat (utility.h:1210) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16E9CA76-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:56.956346] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=4, avg_log_batch_cnt=1, total_group_log_size=911, avg_group_log_size=227) [2025-01-08 00:51:56.956382] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=911) [2025-01-08 00:51:56.956507] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:56.958326] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:56.958400] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4915, iops:5, bw:23468, [delay/us]:prepare:9, schedule:1003, submit:106, rt:159851, total:160970) [2025-01-08 00:51:56.958427] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=1, iops=1, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:56.958555] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=101] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4915, avg_iops:5, avg_bw:23469, [delay/us]: prepare:9, schedule:1003, submit:106, device:159851, total:160970) [2025-01-08 00:51:56.958584] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:56.958659] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, bw:39114, [delay/us]:prepare:9, schedule:1678, submit:821, rt:87071, total:89579) [2025-01-08 00:51:56.958693] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:56.958765] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:39112, [delay/us]: prepare:9, schedule:1678, submit:821, device:87071, total:89579) [2025-01-08 00:51:56.958786] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:56.958859] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:19556, [delay/us]:prepare:6, schedule:73, submit:1, rt:147131, total:147211) [2025-01-08 00:51:56.958945] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=63] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:12288 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:15, bw:62579, [delay/us]:prepare:10, schedule:1545, submit:84786, rt:51859, total:138201) [2025-01-08 00:51:56.958964] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1002, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:56.959050] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=72] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:19555, [delay/us]: prepare:6, schedule:73, submit:1, device:147131, total:147211) [2025-01-08 00:51:56.959082] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:16, avg_bw:62576, [delay/us]: prepare:10, schedule:1545, submit:84786, device:51859, total:138201) [2025-01-08 00:51:56.959115] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:56.959152] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3910, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:372782, total:372791) [2025-01-08 00:51:56.959397] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:56.959462] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3910, [delay/us]: prepare:0, schedule:0, submit:600000000, device:372782, total:372791) [2025-01-08 00:51:56.959491] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:56.959517] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:56.959529] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:56.959543] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:56.959554] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SERVER] net_in=155kB/s, net_out=168kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:56.965323] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84476263}], scn_array=[cnt:1, {val:1736326316939590000, v:0}], curr_size=545, accum_size=1497, time_cost=22287) [2025-01-08 00:51:56.965572] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:56.965617] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=254498, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:56.965630] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=94861, avg_log_gen_to_submit_time=94942, avg_log_submit_to_flush_time=142569, avg_log_submit_to_slide_time=159556) [2025-01-08 00:51:56.965776] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2010, hit_cnt=6, read_cnt=6, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:56.968321] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385720-0-0] [lt=16] InfoPoolResizeTask(cost_ts=7) [2025-01-08 00:51:56.988744] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CED-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:56.988815] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CED-0-0] [lt=46] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=105) [2025-01-08 00:51:57.008077] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50513][T1002_L0_G0][T1002][YB42C0A89F8A-00062B2D1BC9C7E6-0-0] [lt=4] get number(ret=0, service_type_=0, range=1, base_id=1736326317007408000, start_id=1736326317007408000, end_id=1736326317007408001) [2025-01-08 00:51:57.033895] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F5E3-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1736326317032837000, start_id=1736326317032837000, end_id=1736326317032837001) [2025-01-08 00:51:57.035428] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=9] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:57.034998] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58646][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:57.059424] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58646][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A3F-0-0] [lt=501] init point thread id with(&point=0x5572d50e6f00, idx_=1303, point=[thread id=58646, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58646) [2025-01-08 00:51:57.060275] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:57.060582] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [46323][T1_RFLWorker][T1][YB42C0A89F89-00062B2D1618458E-0-0] [lt=27] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:51:57.072252] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get_min_unreplayed_log_info(lsn={lsn:2250008}, scn={val:1736326316274758004, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326317037882, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2250008}, next_to_submit_scn_:{val:1736326316274758004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2249886}, end_lsn:{lsn:2250008}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2250008}, readable_log_tail:{lsn:2250008}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64854760, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326316274758003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326316274758003, v:0}, data_checksum:3348837161, flag:29907}}, init_mode_version:0, accumulate_checksum:941825696, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326316274758003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18392, miss_cnt:0, cache_read_size:2250008, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:57.091560] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:57.091608] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=32] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=96) [2025-01-08 00:51:57.091920] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE3-0-0] [lt=22] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10006, add_timestamp:1736326317091903}) [2025-01-08 00:51:57.092549] WDIAG [SHARE] inner_to_show_str (ob_system_variable.cpp:516) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE3-0-0] [lt=41][errcode=0] sys var casted obj is not ObVarcharType(ret=0, value={"NULL":"NULL"}, *res_obj={"NULL":"NULL"}, get_name()=secure_file_priv) [2025-01-08 00:51:57.092758] WDIAG [SHARE] inner_to_show_str (ob_system_variable.cpp:516) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE3-0-0] [lt=22][errcode=0] sys var casted obj is not ObVarcharType(ret=0, value={"NULL":"NULL"}, *res_obj={"NULL":"NULL"}, get_name()=external_user) [2025-01-08 00:51:57.093673] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A7-0-0] [lt=43] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10006, locations=[{table_id:10006, partition_id:10006, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317093635, sql_renew_time:1736326317093635, is_mark_fail:false}]) [2025-01-08 00:51:57.093728] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A7-0-0] [lt=53] success to process renew task(task={tenant_id:1, table_id:10006, add_timestamp:1736326317091903}, locations=[{table_id:10006, partition_id:10006, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317093635, sql_renew_time:1736326317093635, is_mark_fail:false}]) [2025-01-08 00:51:57.103890] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=22] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84477394}, readable_log_tail:{lsn:84477394}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49732142, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:57.103993] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=101] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:57.104023] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:57.120368] INFO [STORAGE.BLKMGR] mark_macro_blocks (ob_block_manager.cpp:1076) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=260] no block alloc/free, no need to mark blocks(ret=0, mark_info.count()=0) [2025-01-08 00:51:57.120407] INFO [STORAGE.BLKMGR] mark_and_sweep (ob_block_manager.cpp:1056) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=36] finish once mark and sweep(ret=0, alloc_num=2962, mark_info.count()=0, marker_status={total_block_count:1024, reserved_block_count:2, linked_block_count:0, tmp_file_count:0, data_block_count:11, shared_data_block_count:11, index_block_count:0, ids_block_count:0, disk_block_count:0, bloomfiter_count:0, hold_count:41, pending_free_count:1, free_count:981, shared_meta_block_count:19, mark_cost_time:105, sweep_cost_time:0, start_time:"2025-01-08 00:51:57.120299", last_end_time:"2025-01-08 00:20:57.6873", hold_info:nothing, mark_finished:false}, map_cnt=41) [2025-01-08 00:51:57.140043] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=29] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1525616, replace_num=15728) [2025-01-08 00:51:57.140117] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=72] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:57.140479] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:57.144824] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=199, gc_interval=10000000) [2025-01-08 00:51:57.144875] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=28] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:51:57.144892] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=13] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:51:57.145492] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=36] new mysql sessid created(conn.sessid_=3221587620, support_ssl=false) [2025-01-08 00:51:57.145548] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221587620, sess.client_addr_="127.0.0.1:51740") [2025-01-08 00:51:57.145625] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=71] accept one succ(*s={this:0x2ad8efff8030, session_id:3221587620, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:156}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.145797] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221587621, support_ssl=false) [2025-01-08 00:51:57.145847] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221587621, sess.client_addr_="127.0.0.1:51746") [2025-01-08 00:51:57.145862] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x2ad8efff8db0, session_id:3221587621, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:183}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.146275] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] new mysql sessid created(conn.sessid_=3221587622, support_ssl=false) [2025-01-08 00:51:57.146412] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221587622, sess.client_addr_="127.0.0.1:51754") [2025-01-08 00:51:57.146430] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x2ad8efffadb0, session_id:3221587622, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.147114] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=189] new mysql sessid created(conn.sessid_=3221587623, support_ssl=false) [2025-01-08 00:51:57.147185] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=34] sm conn init succ(conn.sessid_=3221587623, sess.client_addr_="127.0.0.1:51728") [2025-01-08 00:51:57.147202] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x2ad8a0dfc030, session_id:3221587623, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:193}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.147401] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=179] new mysql sessid created(conn.sessid_=3221587624, support_ssl=false) [2025-01-08 00:51:57.147448] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221587624, sess.client_addr_="127.0.0.1:51744") [2025-01-08 00:51:57.147463] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x2ad8a0de8030, session_id:3221587624, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:194}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.147490] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221587625, support_ssl=false) [2025-01-08 00:51:57.147520] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221587625, sess.client_addr_="127.0.0.1:51756") [2025-01-08 00:51:57.147531] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0de8db0, session_id:3221587625, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:195}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.147555] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221587626, support_ssl=false) [2025-01-08 00:51:57.147583] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221587626, sess.client_addr_="127.0.0.1:51760") [2025-01-08 00:51:57.147594] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0dea030, session_id:3221587626, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:196}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.147698] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=97] new mysql sessid created(conn.sessid_=3221587627, support_ssl=false) [2025-01-08 00:51:57.147742] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221587627, sess.client_addr_="127.0.0.1:51732") [2025-01-08 00:51:57.147755] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0deadb0, session_id:3221587627, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:197}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.147779] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221587628, support_ssl=false) [2025-01-08 00:51:57.147807] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221587628, sess.client_addr_="127.0.0.1:51738") [2025-01-08 00:51:57.147819] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8a0dec030, session_id:3221587628, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:198}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.147840] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221587629, support_ssl=false) [2025-01-08 00:51:57.147867] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221587629, sess.client_addr_="127.0.0.1:51748") [2025-01-08 00:51:57.147898] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] accept one succ(*s={this:0x2ad8a0decdb0, session_id:3221587629, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:200}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.147932] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221587630, support_ssl=false) [2025-01-08 00:51:57.147981] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221587630, sess.client_addr_="127.0.0.1:51758") [2025-01-08 00:51:57.147995] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x2ad8a0dee030, session_id:3221587630, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:201}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.148065] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=56] new mysql sessid created(conn.sessid_=3221587631, support_ssl=false) [2025-01-08 00:51:57.148105] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221587631, sess.client_addr_="127.0.0.1:51726") [2025-01-08 00:51:57.148117] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efffc030, session_id:3221587631, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:202}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.148139] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221587632, support_ssl=false) [2025-01-08 00:51:57.148167] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221587632, sess.client_addr_="127.0.0.1:51730") [2025-01-08 00:51:57.148240] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=68] accept one succ(*s={this:0x2ad8efffcdb0, session_id:3221587632, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:203}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.148268] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221587633, support_ssl=false) [2025-01-08 00:51:57.148373] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=77] sm conn init succ(conn.sessid_=3221587633, sess.client_addr_="127.0.0.1:51734") [2025-01-08 00:51:57.148387] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efe0a030, session_id:3221587633, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:204}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.148410] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221587634, support_ssl=false) [2025-01-08 00:51:57.148467] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221587634, sess.client_addr_="127.0.0.1:51736") [2025-01-08 00:51:57.148543] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=71] accept one succ(*s={this:0x2ad8efe0adb0, session_id:3221587634, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:205}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.148579] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221587635, support_ssl=false) [2025-01-08 00:51:57.148614] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221587635, sess.client_addr_="127.0.0.1:51742") [2025-01-08 00:51:57.148626] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efe0c030, session_id:3221587635, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:206}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.148648] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221587636, support_ssl=false) [2025-01-08 00:51:57.148677] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221587636, sess.client_addr_="127.0.0.1:51750") [2025-01-08 00:51:57.148688] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efe0cdb0, session_id:3221587636, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:207}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.148709] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221587637, support_ssl=false) [2025-01-08 00:51:57.148737] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221587637, sess.client_addr_="127.0.0.1:51752") [2025-01-08 00:51:57.148807] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=66] accept one succ(*s={this:0x2ad8efe0e030, session_id:3221587637, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:208}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.150293] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE7-0-0] [lt=34] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12276, add_timestamp:1736326317150205}) [2025-01-08 00:51:57.151068] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A8-0-0] [lt=20] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12276, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:57.151437] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.151613] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=90] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.151682] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.151717] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.151736] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.151771] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.152295] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6B-0-0] [lt=26] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.152723] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8C-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.153634] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6B-0-0] [lt=21] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587622, proxy_sessid=0, client_sessid=3221587622, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.153810] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6C-0-0] [lt=29] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.153980] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8C-0-0] [lt=80] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587621, proxy_sessid=0, client_sessid=3221587621, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.154719] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6C-0-0] [lt=15] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587620, proxy_sessid=0, client_sessid=3221587620, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.155361] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A8-0-0] [lt=28] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12276, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317155336, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317155336, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317155336, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.155406] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A8-0-0] [lt=45] success to process renew task(task={tenant_id:1, table_id:12276, add_timestamp:1736326317150205}, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317155336, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317155336, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317155336, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.155691] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:51:57.156950] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=40] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:51:57.156977] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=22] gc_check_member_list_ cost time(ret=0, time_us=12124) [2025-01-08 00:51:57.156994] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:51:57.158551] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:57.158957] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159068] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159096] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159226] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=52] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159262] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159296] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159318] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159336] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159350] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159369] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159382] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159398] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159411] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159428] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159465] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159506] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=26] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159523] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159540] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159554] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159569] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159583] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159598] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159623] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159643] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159657] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159674] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159687] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159702] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.159715] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:51:57.159734] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:51:57.160143] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6D-0-0] [lt=43] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.160417] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=20] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:57.161521] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6D-0-0] [lt=17] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587637, proxy_sessid=0, client_sessid=3221587637, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.161609] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6E-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.161768] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:57.162356] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6E-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587636, proxy_sessid=0, client_sessid=3221587636, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.162436] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6F-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.163103] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A6F-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587635, proxy_sessid=0, client_sessid=3221587635, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.163181] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A70-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.163503] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=31] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:51:57.163530] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=21] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:57.163552] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=20] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:57.163564] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=12] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:57.163575] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:51:57.163586] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=58) [2025-01-08 00:51:57.163595] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:51:57.163963] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A70-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587634, proxy_sessid=0, client_sessid=3221587634, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.164045] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A71-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.164812] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A71-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587633, proxy_sessid=0, client_sessid=3221587633, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.164890] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A72-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.164924] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46507][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D0-0-0] [lt=79] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10008, add_timestamp:1736326317164906}) [2025-01-08 00:51:57.164978] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46507][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D0-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12358, add_timestamp:1736326317164976}) [2025-01-08 00:51:57.164997] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46507][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D0-0-0] [lt=11] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12396, add_timestamp:1736326317164996}) [2025-01-08 00:51:57.165735] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A72-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587632, proxy_sessid=0, client_sessid=3221587632, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.165872] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A73-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.166768] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A73-0-0] [lt=17] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587631, proxy_sessid=0, client_sessid=3221587631, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.166851] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A74-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.167642] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A74-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587630, proxy_sessid=0, client_sessid=3221587630, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.167731] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A75-0-0] [lt=24] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.168525] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A75-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587629, proxy_sessid=0, client_sessid=3221587629, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.168605] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A76-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.169134] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46507][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D0-0-0] [lt=9][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:57.169250] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A76-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587628, proxy_sessid=0, client_sessid=3221587628, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.169347] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A77-0-0] [lt=37] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.170086] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A77-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587627, proxy_sessid=0, client_sessid=3221587627, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.170165] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A78-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.170584] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:57.170608] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=31] LogLoopThread round_cost_time(us)(round_cost_time=16) [2025-01-08 00:51:57.170835] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=119] get_max_applied_scn(scn={val:1736326317007408000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18737491}, palf_committed_end_scn_:{val:1736326317007408000, v:0}, last_check_scn_:{val:1736326317007408001, v:0}, max_applied_cb_scn_:{val:1736326317007408000, v:0}}) [2025-01-08 00:51:57.170862] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18737491}, readable_log_tail:{lsn:18737491}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48367277, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:57.170958] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=75] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:57.171023] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A88-0-0] [lt=99] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.171794] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A88-0-0] [lt=19] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587625, proxy_sessid=0, client_sessid=3221587625, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.171896] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A89-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.172654] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] no need to update min/max version(new_version={val:1736326315849992001, v:0}, min_version={val:1736326315849992001, v:0}, max_version={val:1736326315849992001, v:0}) [2025-01-08 00:51:57.172736] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A89-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587624, proxy_sessid=0, client_sessid=3221587624, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.172864] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8A-0-0] [lt=24] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:51:57.173644] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=62] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:57.173909] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8A-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587623, proxy_sessid=0, client_sessid=3221587623, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.174721] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A9-0-0] [lt=26] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=10008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:57.176094] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A9-0-0] [lt=20] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10008, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176070, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176070, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176070, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.176139] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859A9-0-0] [lt=44] success to process renew task(task={tenant_id:1, table_id:10008, add_timestamp:1736326317164906}, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176070, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176070, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176070, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.176181] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AA-0-0] [lt=26] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12358, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176174, sql_renew_time:1736326317176174, is_mark_fail:false}]) [2025-01-08 00:51:57.176439] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AA-0-0] [lt=256] success to process renew task(task={tenant_id:1, table_id:12358, add_timestamp:1736326317164976}, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176174, sql_renew_time:1736326317176174, is_mark_fail:false}]) [2025-01-08 00:51:57.176654] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AB-0-0] [lt=189] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12396, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176641, sql_renew_time:1736326317176641, is_mark_fail:false}]) [2025-01-08 00:51:57.176671] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AB-0-0] [lt=18] success to process renew task(task={tenant_id:1, table_id:12396, add_timestamp:1736326317164996}, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176641, sql_renew_time:1736326317176641, is_mark_fail:false}]) [2025-01-08 00:51:57.176882] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffadb0, session_id:3221587622, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326317157901, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.176912] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221587622, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.177039] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=125][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8db0, session_id:3221587621, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326317157901, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.177066] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221587621, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.177077] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8030, session_id:3221587620, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326317157901, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.177098] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221587620, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.177106] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8efffadb0, session_id:3221587622, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326317157901, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.177148] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221587622, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.177320] INFO [SERVER] run (obmp_disconnect.cpp:83) [46493][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=139] free session successfully(sessid=3221587622) [2025-01-08 00:51:57.177345] INFO [SERVER] run (obmp_disconnect.cpp:90) [46493][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=24] mark session id unused(sessid=3221587622) [2025-01-08 00:51:57.177313] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221587622, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857D1-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:368, rec_:368, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.177442] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=125] sock ref clean, do destroy(*s={this:0x2ad8efff8db0, session_id:3221587621, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:183}, err:5, last_decode_time:1736326317157901, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.177481] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] unlock session of tenant(conn.sessid_=3221587621, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.177490] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221587621, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857D0-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:436, rec_:436, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.177520] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] sock ref clean, do destroy(*s={this:0x2ad8efff8030, session_id:3221587620, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326317157901, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.177564] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221587620, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.177576] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221587620, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857CF-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:134, rec_:134, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:7}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.177666] INFO [SERVER] run (obmp_disconnect.cpp:83) [46496][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=28] free session successfully(sessid=3221587621) [2025-01-08 00:51:57.177684] INFO [SERVER] run (obmp_disconnect.cpp:90) [46496][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=17] mark session id unused(sessid=3221587621) [2025-01-08 00:51:57.177794] INFO [SERVER] run (obmp_disconnect.cpp:83) [46496][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=58] free session successfully(sessid=3221587620) [2025-01-08 00:51:57.177807] INFO [SERVER] run (obmp_disconnect.cpp:90) [46496][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=13] mark session id unused(sessid=3221587620) [2025-01-08 00:51:57.177968] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB7-0-0] [lt=5] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11021, add_timestamp:1736326317177954}) [2025-01-08 00:51:57.178613] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB8-0-0] [lt=7] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11003, add_timestamp:1736326317178609}) [2025-01-08 00:51:57.183675] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB9-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:57.183833] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB9-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:57.186299] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B9B-0-0] [lt=24] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11018, add_timestamp:1736326317186291}) [2025-01-08 00:51:57.188013] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_stat.cpp:301) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AB7-0-0] [lt=15][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:57.189703] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AC-0-0] [lt=164] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11021, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:57.190839] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46492][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B9C-0-0] [lt=33][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:57.192751] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=52] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:57.193266] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=173] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12142, add_timestamp:1736326317193207}) [2025-01-08 00:51:57.193312] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=28] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12072, add_timestamp:1736326317193310}) [2025-01-08 00:51:57.193330] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=11] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12084, add_timestamp:1736326317193329}) [2025-01-08 00:51:57.193346] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12068, add_timestamp:1736326317193345}) [2025-01-08 00:51:57.203158] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B9D-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:57.203425] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEA-0-0] [lt=26] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11037, add_timestamp:1736326317203418}) [2025-01-08 00:51:57.203922] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEB-0-0] [lt=109] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12325, add_timestamp:1736326317203918}) [2025-01-08 00:51:57.204020] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEB-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1736326317192724000, start_id=1736326317192724000, end_id=1736326317192724001) [2025-01-08 00:51:57.207030] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEC-0-0] [lt=27] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11008, add_timestamp:1736326317207026}) [2025-01-08 00:51:57.211973] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=18] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12217, add_timestamp:1736326317211967}) [2025-01-08 00:51:57.212009] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=21][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12142, add_timestamp:1736326317212006}) [2025-01-08 00:51:57.212021] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=13][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12142, add_timestamp:1736326317212006}) [2025-01-08 00:51:57.212030] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=8][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=12142, vtable_locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227357, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.212060] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=23] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11035, add_timestamp:1736326317212059}) [2025-01-08 00:51:57.212075] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=8][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12072, add_timestamp:1736326317212074}) [2025-01-08 00:51:57.212083] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=8][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12072, add_timestamp:1736326317212074}) [2025-01-08 00:51:57.212090] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=6][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=12072, vtable_locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227465, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.212130] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=28][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12068, add_timestamp:1736326317212128}) [2025-01-08 00:51:57.212144] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=14][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12068, add_timestamp:1736326317212128}) [2025-01-08 00:51:57.212155] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46495][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CED-0-0] [lt=10][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=12068, vtable_locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302227502, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.214524] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEE-0-0] [lt=41] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11017, add_timestamp:1736326317214521}) [2025-01-08 00:51:57.275467] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:57.275466] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEF-0-0] [lt=123][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326317275458}) [2025-01-08 00:51:57.275505] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEF-0-0] [lt=38][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326317275458}) [2025-01-08 00:51:57.275516] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEF-0-0] [lt=9][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=11017, vtable_locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302226658, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302226658, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326302226658, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.275549] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4015] clock out of order(cur_ts=1736326317275547, cur_ts_=1736326317192724, delta=82823) [2025-01-08 00:51:57.276279] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:57.276307] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=13] ====== traversal_flush timer task ====== [2025-01-08 00:51:57.276394] INFO [MDS] flush (mds_table_mgr.cpp:195) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853FC-0-0] [lt=10] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326317007408000, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326330087570000, latest_log_ts:{val:1736326313644270001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:57.276536] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853FC-0-0] [lt=98] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326312645069002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:57.276649] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853FC-0-0] [lt=84] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326312811284000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:57.276733] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853FC-0-0] [lt=57] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326312813632000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:57.276853] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D1853FC-0-0] [lt=116] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:51:57.276976] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] no need to update min/max version(new_version={val:1736326315849992001, v:0}, min_version={val:1736326315849992001, v:0}, max_version={val:1736326315849992001, v:0}) [2025-01-08 00:51:57.277153] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=47] ====== check clog disk timer task ====== [2025-01-08 00:51:57.277182] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:57.277360] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=6239232) [2025-01-08 00:51:57.277839] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:51:57.278456] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:57.278606] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326316274758001, v:0}, base_version={val:1736326316274758001, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326316274758003, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326317278588, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326316274758001, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326317167645, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326316274758001, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326317205566, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:51:57.278656] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=48] no need to update min/max version(new_version={val:1736326316274758001, v:0}, min_version={val:1736326316274758001, v:0}, max_version={val:1736326316274758001, v:0}) [2025-01-08 00:51:57.282862] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2237600}], scn_array=[cnt:1, {val:1736326317048856001, v:0}], curr_size=122, accum_size=610, time_cost=68571) [2025-01-08 00:51:57.282959] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=94] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=610) [2025-01-08 00:51:57.283051] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0de8db0, session_id:3221587625, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326317185465, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.283178] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=116] kill and revert session(conn.sessid_=3221587625, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.283252] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=72] sock ref clean, do destroy(*s={this:0x2ad8a0de8db0, session_id:3221587625, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326317185465, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.283296] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] unlock session of tenant(conn.sessid_=3221587625, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.283308] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221587625, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B9D-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:633, rec_:633, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.283642] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=25] free session successfully(sessid=3221587625) [2025-01-08 00:51:57.283668] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=25] mark session id unused(sessid=3221587625) [2025-01-08 00:51:57.283666] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEF-0-0] [lt=34][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:57.283750] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe0adb0, session_id:3221587634, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:205}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.283824] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221587634, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.283846] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffcdb0, session_id:3221587632, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:292, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.283872] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221587632, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.283882] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] sock ref clean, do destroy(*s={this:0x2ad8efe0adb0, session_id:3221587634, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:205}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.283922] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221587634, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.283932] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221587634, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CEC-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:133, rec_:133, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:31, seq_:37}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:38}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.283963] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] sock ref clean, do destroy(*s={this:0x2ad8efffcdb0, session_id:3221587632, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:292, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.284049] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221587632, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.284062] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221587632, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CEA-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:187, rec_:187, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:2, seq_:4}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:5}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:6}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.284244] INFO [SERVER] run (obmp_disconnect.cpp:83) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=42] free session successfully(sessid=3221587634) [2025-01-08 00:51:57.284289] INFO [SERVER] run (obmp_disconnect.cpp:90) [46497][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=43] mark session id unused(sessid=3221587634) [2025-01-08 00:51:57.284685] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58615][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48C5-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=6, avg_log_batch_cnt=0, total_group_log_size=732, avg_group_log_size=122) [2025-01-08 00:51:57.284750] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58615][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48C5-0-0] [lt=62] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=732) [2025-01-08 00:51:57.288967] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.288994] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=26][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289021] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=14][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289029] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289043] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289050] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289063] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289070] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289082] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289089] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289101] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289108] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289120] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289127] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289140] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289146] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289159] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289165] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289295] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=114][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289304] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=9][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289317] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289324] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289336] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289343] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289355] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289362] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289374] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289381] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289393] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289400] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289412] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289418] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289430] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289437] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289449] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289456] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289468] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289474] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289551] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=63][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289559] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289551] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=133] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326313162481, last_first_register_time_us=1736324449183670, register_time_us=1736326313162481, curr_time_us=1736326317289549) [2025-01-08 00:51:57.289611] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326317289581, last_submit_register_req_time_us=1736326317289581, register_parent_reason=1) [2025-01-08 00:51:57.289620] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289627] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:51:57.289633] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=13][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289652] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=9][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289660] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289673] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289679] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289692] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289698] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289710] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289717] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289729] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289736] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289759] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289772] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=13][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289794] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=11][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289804] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=10][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289818] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289825] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289837] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289844] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289856] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289862] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289875] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289881] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.289893] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.289986] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=92][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290008] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=10][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290016] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290029] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290036] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290048] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290055] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290067] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290074] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290086] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290092] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290105] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290111] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290123] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290130] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290142] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290149] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290161] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290168] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290180] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290187] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290262] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290275] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=13][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290294] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=9][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290301] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290313] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290320] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290332] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290352] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=19][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290376] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=13][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290386] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=10][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290404] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=11][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290414] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=10][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290431] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=11][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290438] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290450] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290456] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290469] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290476] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290533] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290546] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=13][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290562] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=9][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290569] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290582] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290588] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290601] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290607] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290620] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290626] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290639] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290645] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290658] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290664] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290677] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290683] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290696] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290702] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290715] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290721] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290733] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290740] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290752] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290759] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290771] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290778] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290790] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290796] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290861] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290873] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=11][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290890] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=8][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290898] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290910] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290917] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=6][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290929] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290936] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] string accuracy check failed(cast_ret=-5167, ret=0, warning=-5167, is_oracle=false, meta={type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}) [2025-01-08 00:51:57.290948] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12264) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] char type length is too long(max_accuracy_len=8, trunc_len_char=11) [2025-01-08 00:51:57.290954] WDIAG [SQL] string_length_check (ob_datum_cast.cpp:12277) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CE9-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:57.296904] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C89C2B7-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326317295370000, start_id=1736326317295370000, end_id=1736326317295370001) [2025-01-08 00:51:57.297482] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:57.297710] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1736326317297707, bytes: 0, bw: 0.000000 MB/s, add_ts: 1015854, add_bytes: 0 [2025-01-08 00:51:57.297743] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AC-0-0] [lt=25] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11021, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317297716, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317297716, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317297716, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.297792] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AC-0-0] [lt=48] success to process renew task(task={tenant_id:1, table_id:11021, add_timestamp:1736326317177954}, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317297716, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317297716, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317297716, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.297833] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AD-0-0] [lt=26] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11003, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:57.298614] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58638][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=38] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:51:57.299981] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=16] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:196K/44:2634726 disk:196K/44:2634726 t1m2:28K/6:1912358 t1m2g0:28K/6:1912358 t1002m2:116K/27:3303419 t1002m2g0:116K/27:3303419 t1001m2:52K/11:1382827 t1001m2g0:52K/11:1382827 |q0: 0 1 0 |q1: 0 0 0 [2025-01-08 00:51:57.300062] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=19] ====== check clog disk timer task ====== [2025-01-08 00:51:57.300090] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:57.300219] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:57.314584] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58641][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DADD-0-0] [lt=57][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18384, log_task={header:{begin_lsn:{lsn:2237600}, end_lsn:{lsn:2237722}, log_id:18384, min_scn:{val:1736326317048856001, v:0}, max_scn:{val:1736326317048856001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2237478}, prev_proposal_id:1, committed_end_lsn:{lsn:2237600}, data_checksum:940009244, accum_checksum:406732393, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326317197124, freeze_ts:1736326317197124, submit_ts:1736326317197136, flushed_ts:1736326317297466, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:100330}, fs_cb_begin_ts=1736326317314551, log_life_time=117427) [2025-01-08 00:51:57.314699] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A967C-0-0] [lt=48] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=854) [2025-01-08 00:51:57.314924] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A0D-0-0] [lt=35][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=45099, log_task={header:{begin_lsn:{lsn:84477394}, end_lsn:{lsn:84477516}, log_id:45099, min_scn:{val:1736326316967049001, v:0}, max_scn:{val:1736326316967049001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84477394}, data_checksum:667309872, accum_checksum:630276418, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326317159876, freeze_ts:1736326317159885, submit_ts:1736326317159956, flushed_ts:1736326317175144, gen_to_freeze cost time:9, gen_to_submit cost time:80, submit_to_flush cost time:15188}, fs_cb_begin_ts=1736326317314897, log_life_time=155021) [2025-01-08 00:51:57.317124] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=126][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43610658}], scn_array=[cnt:1, {val:1736326317295370000, v:0}], curr_size=546, accum_size=7650, time_cost=17326) [2025-01-08 00:51:57.317156] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=7650) [2025-01-08 00:51:57.317676] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=132] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=7693) [2025-01-08 00:51:57.317795] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58644][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3BDC-0-0] [lt=33] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=48063, avg_log_gen_to_submit_time=48207, avg_log_submit_to_flush_time=15780, avg_log_submit_to_slide_time=23645) [2025-01-08 00:51:57.318089] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=65][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2216250}], scn_array=[cnt:1, {val:1736326317157901000, v:0}], curr_size=122, accum_size=854, time_cost=20209) [2025-01-08 00:51:57.318213] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=123] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=854) [2025-01-08 00:51:57.318321] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18737613}], scn_array=[cnt:1, {val:1736326317157901000, v:0}], curr_size=122, accum_size=1700, time_cost=20475) [2025-01-08 00:51:57.319476] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:57.319527] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:57.321486] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AD-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11003, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317321457, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317321457, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317321457, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.321624] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AD-0-0] [lt=136] success to process renew task(task={tenant_id:1, table_id:11003, add_timestamp:1736326317178609}, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317321457, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317321457, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317321457, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.321668] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AE-0-0] [lt=27] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11018, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:57.326076] INFO [SERVER] run (obmp_disconnect.cpp:83) [46500][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=116] free session successfully(sessid=3221587632) [2025-01-08 00:51:57.326105] INFO [SERVER] run (obmp_disconnect.cpp:90) [46500][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=27] mark session id unused(sessid=3221587632) [2025-01-08 00:51:57.328737] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326316977540000, v:0}, new_max_version={val:1736326316977540000, v:0}, last_min_version={val:1736326316677040001, v:0}, last_max_version={val:1736326316677040001, v:0}, total_time_us=30724, delta_us=351195) [2025-01-08 00:51:57.328846] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326316977540000, v:0}, min_version={val:1736326316977540000, v:0}, max_version={val:1736326316977540000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:57.330663] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AE-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11018, locations=[{table_id:11018, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317330281, sql_renew_time:0, is_mark_fail:false}, {table_id:11018, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317330281, sql_renew_time:0, is_mark_fail:false}, {table_id:11018, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317330281, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.330717] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AE-0-0] [lt=53] success to process renew task(task={tenant_id:1, table_id:11018, add_timestamp:1736326317186291}, locations=[{table_id:11018, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317330281, sql_renew_time:0, is_mark_fail:false}, {table_id:11018, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317330281, sql_renew_time:0, is_mark_fail:false}, {table_id:11018, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317330281, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.330796] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AF-0-0] [lt=45] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12142, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317330782, sql_renew_time:1736326317330782, is_mark_fail:false}]) [2025-01-08 00:51:57.331030] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859AF-0-0] [lt=232] success to process renew task(task={tenant_id:1, table_id:12142, add_timestamp:1736326317193207}, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317330782, sql_renew_time:1736326317330782, is_mark_fail:false}]) [2025-01-08 00:51:57.331070] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B0-0-0] [lt=21] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12072, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317331060, sql_renew_time:1736326317331060, is_mark_fail:false}]) [2025-01-08 00:51:57.331149] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B0-0-0] [lt=79] success to process renew task(task={tenant_id:1, table_id:12072, add_timestamp:1736326317193310}, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317331060, sql_renew_time:1736326317331060, is_mark_fail:false}]) [2025-01-08 00:51:57.331179] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B1-0-0] [lt=13] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12084, locations=[{table_id:12084, partition_id:12084, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317331170, sql_renew_time:1736326317331170, is_mark_fail:false}]) [2025-01-08 00:51:57.331376] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B1-0-0] [lt=194] success to process renew task(task={tenant_id:1, table_id:12084, add_timestamp:1736326317193329}, locations=[{table_id:12084, partition_id:12084, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317331170, sql_renew_time:1736326317331170, is_mark_fail:false}]) [2025-01-08 00:51:57.331687] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B2-0-0] [lt=282] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12068, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317331672, sql_renew_time:1736326317331672, is_mark_fail:false}]) [2025-01-08 00:51:57.331721] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B2-0-0] [lt=33] success to process renew task(task={tenant_id:1, table_id:12068, add_timestamp:1736326317193345}, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317331672, sql_renew_time:1736326317331672, is_mark_fail:false}]) [2025-01-08 00:51:57.331758] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B3-0-0] [lt=25] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11037, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:57.333704] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A78-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221587626, proxy_sessid=0, client_sessid=3221587626, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:51:57.335811] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B3-0-0] [lt=18] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11037, locations=[{table_id:11037, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317335794, sql_renew_time:0, is_mark_fail:false}, {table_id:11037, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317335794, sql_renew_time:0, is_mark_fail:false}, {table_id:11037, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317335794, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.335853] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B3-0-0] [lt=41] success to process renew task(task={tenant_id:1, table_id:11037, add_timestamp:1736326317203418}, locations=[{table_id:11037, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317335794, sql_renew_time:0, is_mark_fail:false}, {table_id:11037, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317335794, sql_renew_time:0, is_mark_fail:false}, {table_id:11037, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317335794, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.335889] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B4-0-0] [lt=25] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12325, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:57.336768] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=27] start to print blacklist info [2025-01-08 00:51:57.337139] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=24] ls blacklist refresh finish(cost_time=47061) [2025-01-08 00:51:57.337183] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2250130}], scn_array=[cnt:1, {val:1736326317032837001, v:0}], curr_size=122, accum_size=732, time_cost=39767) [2025-01-08 00:51:57.337330] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=145] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=732) [2025-01-08 00:51:57.339656] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B4-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12325, locations=[{table_id:12325, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317339634, sql_renew_time:0, is_mark_fail:false}, {table_id:12325, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317339634, sql_renew_time:0, is_mark_fail:false}, {table_id:12325, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317339634, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.339701] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B4-0-0] [lt=44] success to process renew task(task={tenant_id:1, table_id:12325, add_timestamp:1736326317203918}, locations=[{table_id:12325, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317339634, sql_renew_time:0, is_mark_fail:false}, {table_id:12325, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317339634, sql_renew_time:0, is_mark_fail:false}, {table_id:12325, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317339634, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.339963] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B5-0-0] [lt=247] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:57.346048] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEB-0-0] [lt=17][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:51:57.350846] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:57.351125] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=13] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6708097024, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:57.353930] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B5-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11008, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317353900, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317353900, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317353900, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.353986] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B5-0-0] [lt=54] success to process renew task(task={tenant_id:1, table_id:11008, add_timestamp:1736326317207026}, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317353900, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317353900, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317353900, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.354039] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B6-0-0] [lt=29] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12217, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354031, sql_renew_time:1736326317354031, is_mark_fail:false}]) [2025-01-08 00:51:57.354055] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B6-0-0] [lt=16] success to process renew task(task={tenant_id:1, table_id:12217, add_timestamp:1736326317211967}, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354031, sql_renew_time:1736326317354031, is_mark_fail:false}]) [2025-01-08 00:51:57.354076] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B7-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11035, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354072, sql_renew_time:1736326317354072, is_mark_fail:false}]) [2025-01-08 00:51:57.354090] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B7-0-0] [lt=14] success to process renew task(task={tenant_id:1, table_id:11035, add_timestamp:1736326317212059}, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354072, sql_renew_time:1736326317354072, is_mark_fail:false}]) [2025-01-08 00:51:57.354108] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B8-0-0] [lt=12] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11017, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:51:57.354970] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B8-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11017, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354955, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354955, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354955, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.355011] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B8-0-0] [lt=40] success to process renew task(task={tenant_id:1, table_id:11017, add_timestamp:1736326317214521}, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354955, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354955, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354955, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:51:57.382035] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:57.382590] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] no need to update min/max version(new_version={val:1736326315849992001, v:0}, min_version={val:1736326315849992001, v:0}, max_version={val:1736326315849992001, v:0}) [2025-01-08 00:51:57.385051] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=91][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:57.385104] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=44] kill and revert session(conn.sessid_=3221587624, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.385119] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=1] kill and revert session(conn.sessid_=3221587623, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.385147] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8a0de8030, session_id:3221587624, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326317185465, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.385182] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221587624, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.385194] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221587624, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B9C-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.385284] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=85] sock ref clean, do destroy(*s={this:0x2ad8a0dfc030, session_id:3221587623, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326317185465, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:291, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.385320] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221587623, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.385329] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221587623, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B9B-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:186, rec_:186, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:5, seq_:7}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.385371] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:57.385392] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221587628, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.385402] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=0] kill and revert session(conn.sessid_=3221587630, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.385410] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=0] kill and revert session(conn.sessid_=3221587629, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.385419] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=0] kill and revert session(conn.sessid_=3221587627, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.385425] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8a0dec030, session_id:3221587628, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326317176165, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.385456] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221587628, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.385464] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221587628, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AB7-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:688, rec_:688, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:219, seq_:225}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:226}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.385491] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] sock ref clean, do destroy(*s={this:0x2ad8a0dee030, session_id:3221587630, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326317176165, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.385518] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221587630, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.385525] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221587630, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AB9-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.385551] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] sock ref clean, do destroy(*s={this:0x2ad8a0decdb0, session_id:3221587629, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326317176165, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.385595] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221587629, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.385606] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221587629, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AB8-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:98, rec_:98, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:10}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:11}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.385633] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] sock ref clean, do destroy(*s={this:0x2ad8a0deadb0, session_id:3221587627, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326317176165, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:617, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.385667] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221587627, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.385675] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221587627, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AB6-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:512, rec_:512, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.385713] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:57.385737] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221587637, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.385745] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8efe0e030, session_id:3221587637, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:208}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.385789] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] unlock session of tenant(conn.sessid_=3221587637, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.385801] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221587637, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CEF-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:549, rec_:549, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:12, seq_:19}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:20}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.386193] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=60] free session successfully(sessid=3221587624) [2025-01-08 00:51:57.386209] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=14] mark session id unused(sessid=3221587624) [2025-01-08 00:51:57.386357] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221587623) [2025-01-08 00:51:57.386387] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=30] mark session id unused(sessid=3221587623) [2025-01-08 00:51:57.386445] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=7] free session successfully(sessid=3221587628) [2025-01-08 00:51:57.386456] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221587628) [2025-01-08 00:51:57.386499] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221587630) [2025-01-08 00:51:57.386509] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221587630) [2025-01-08 00:51:57.386560] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221587629) [2025-01-08 00:51:57.386574] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=14] mark session id unused(sessid=3221587629) [2025-01-08 00:51:57.386634] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=7] free session successfully(sessid=3221587627) [2025-01-08 00:51:57.386646] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=12] mark session id unused(sessid=3221587627) [2025-01-08 00:51:57.386689] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221587637) [2025-01-08 00:51:57.386699] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221587637) [2025-01-08 00:51:57.387957] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B9E-0-0] [lt=5] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:57.388006] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46501][T1_L0_G0][T1001][YB42C0A89F89-00062B2D1B484B9E-0-0] [lt=15] get_stable_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:57.388030] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46501][T1_L0_G0][T1002][YB42C0A89F89-00062B2D1B484B9E-0-0] [lt=9] get_stable_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:57.390150] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48C7-0-0] [lt=46] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=21065, avg_log_submit_to_slide_time=66851) [2025-01-08 00:51:57.390283] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48C7-0-0] [lt=130][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18487, log_task={header:{begin_lsn:{lsn:2250130}, end_lsn:{lsn:2250252}, log_id:18487, min_scn:{val:1736326317032837001, v:0}, max_scn:{val:1736326317032837001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2250008}, prev_proposal_id:1, committed_end_lsn:{lsn:2250130}, data_checksum:1715074492, accum_checksum:2211119512, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326317284660, freeze_ts:1736326317284661, submit_ts:1736326317284675, flushed_ts:1736326317337383, gen_to_freeze cost time:1, gen_to_submit cost time:15, submit_to_flush cost time:52708}, fs_cb_begin_ts=1736326317390061, log_life_time=105401) [2025-01-08 00:51:57.391471] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46501][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B9E-0-0] [lt=8][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:57.392415] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=854, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:57.392534] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=110] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18394, miss_cnt:0, cache_read_size:2250252, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:57.393907] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4015] clock out of order(cur_ts=1736326317393904, cur_ts_=1736326317364940, delta=28964) [2025-01-08 00:51:57.395477] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=2] kill and revert session(conn.sessid_=3221587626, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.395502] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] sock ref clean, do destroy(*s={this:0x2ad8a0dea030, session_id:3221587626, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326317364940, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.395543] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] unlock session of tenant(conn.sessid_=3221587626, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.395554] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221587626, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484B9E-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:562, rec_:562, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.395745] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221587626) [2025-01-08 00:51:57.395763] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=18] mark session id unused(sessid=3221587626) [2025-01-08 00:51:57.411901] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=108][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18737735}], scn_array=[cnt:1, {val:1736326317157901001, v:0}], curr_size=122, accum_size=1822, time_cost=19141) [2025-01-08 00:51:57.411990] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=74][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84477516}], scn_array=[cnt:1, {val:1736326317192724000, v:0}], curr_size=122, accum_size=2327, time_cost=17811) [2025-01-08 00:51:57.412017] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2327) [2025-01-08 00:51:57.413051] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2216372}], scn_array=[cnt:1, {val:1736326317157901001, v:0}], curr_size=122, accum_size=122, time_cost=20273) [2025-01-08 00:51:57.438139] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=42] ====== traversal_flush timer task ====== [2025-01-08 00:51:57.438420] INFO [MDS] flush (mds_table_mgr.cpp:195) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E75-0-0] [lt=34] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326317364940000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329284252000, latest_log_ts:{val:1736326312645069001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:57.438529] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E75-0-0] [lt=103] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:57.438684] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2237722}], scn_array=[cnt:1, {val:1736326317048856002, v:0}], curr_size=122, accum_size=122, time_cost=27090) [2025-01-08 00:51:57.451293] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=155] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:57.462593] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:57.493488] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:57.493946] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:57.494964] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] no need to update min/max version(new_version={val:1736326315849992001, v:0}, min_version={val:1736326315849992001, v:0}, max_version={val:1736326315849992001, v:0}) [2025-01-08 00:51:57.496570] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=12] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:51:57.496595] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=21] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=133, cache_node_num=133) [2025-01-08 00:51:57.496608] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=133, cache_node_num=133) [2025-01-08 00:51:57.496620] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:57.499283] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=133, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}, {obj_id:2083, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221102}]) [2025-01-08 00:51:57.499717] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=430] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:57.499749] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] ====== tenant freeze timer task ====== [2025-01-08 00:51:57.499766] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70704720, tenant_id=1001) [2025-01-08 00:51:57.499786] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:57.499827] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=56] ====== tenant freeze timer task ====== [2025-01-08 00:51:57.500568] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B69-0-0] [lt=27][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18385, log_task={header:{begin_lsn:{lsn:2237722}, end_lsn:{lsn:2237844}, log_id:18385, min_scn:{val:1736326317048856002, v:0}, max_scn:{val:1736326317048856002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2237600}, prev_proposal_id:1, committed_end_lsn:{lsn:2237722}, data_checksum:940009244, accum_checksum:3941831920, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326317396083, freeze_ts:1736326317396083, submit_ts:1736326317396141, flushed_ts:1736326317439141, gen_to_freeze cost time:0, gen_to_submit cost time:58, submit_to_flush cost time:43000}, fs_cb_begin_ts=1736326317500557, log_life_time=104474) [2025-01-08 00:51:57.504175] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2254470, this={dio_aligned_buf:{buf_write_offset:1792, buf_padding_size:2304, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:27117, truncate_used_ts:5526}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2252800, buf_len=122, total_write_size=10717, total_write_size_after_dio=372736, ob_pwrite_used_ts=841855, count=88, each_pwrite_cost=9566) [2025-01-08 00:51:57.504286] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=105][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2250374}], scn_array=[cnt:1, {val:1736326317032837003, v:0}], curr_size=122, accum_size=244, time_cost=19306) [2025-01-08 00:51:57.511347] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84477638}], scn_array=[cnt:1, {val:1736326317192724001, v:0}], curr_size=122, accum_size=122, time_cost=16264) [2025-01-08 00:51:57.541433] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:57.567526] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE950-0-0] [lt=47] get number(ret=0, service_type_=0, range=1, base_id=1736326317454295000, start_id=1736326317454295001, end_id=1736326317454295002) [2025-01-08 00:51:57.579637] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48CB-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:57.581575] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=33633, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:57.581618] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2237844}], scn_array=[cnt:1, {val:1736326317048856003, v:0}], curr_size=122, accum_size=244, time_cost=80909) [2025-01-08 00:51:57.581665] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=5072, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:57.581837] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=21037, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:57.581855] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2216494}], scn_array=[cnt:1, {val:1736326317157901002, v:0}], curr_size=122, accum_size=244, time_cost=81947) [2025-01-08 00:51:57.581874] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=3922, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:57.581897] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:57.582751] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:57.588444] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:57.588523] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:51:57.588880] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6705999872, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:57.590632] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=86] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1384108, clean_num=31457) [2025-01-08 00:51:57.590793] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:57.591011] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=59, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:57.591026] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4015] clock out of order(cur_ts=1736326317591023, cur_ts_=1736326317511489, delta=79534) [2025-01-08 00:51:57.591092] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:57.591107] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=35903, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:57.591136] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=47, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:57.591393] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=18791, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:57.591417] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18737857}], scn_array=[cnt:1, {val:1736326317157901002, v:0}], curr_size=122, accum_size=1944, time_cost=91528) [2025-01-08 00:51:57.591445] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2121, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:57.591486] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:57.591496] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=28368, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:57.591509] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=6] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:57.595097] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58644][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=23] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:51:57.596321] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEE-0-0] [lt=24][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:51:57.597768] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=99][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:57.598653] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] no need to update min/max version(new_version={val:1736326315849992001, v:0}, min_version={val:1736326315849992001, v:0}, max_version={val:1736326315849992001, v:0}) [2025-01-08 00:51:57.600562] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=2] kill and revert session(conn.sessid_=3221587636, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.600605] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] sock ref clean, do destroy(*s={this:0x2ad8efe0cdb0, session_id:3221587636, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:207}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.600654] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] unlock session of tenant(conn.sessid_=3221587636, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.600666] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221587636, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CEE-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:241, rec_:241, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.600835] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221587636) [2025-01-08 00:51:57.600850] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=14] mark session id unused(sessid=3221587636) [2025-01-08 00:51:57.604346] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D3-0-0] [lt=59] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326317619047, keepalive_ts:-1}) [2025-01-08 00:51:57.604374] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D3-0-0] [lt=26] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326317619047, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:57.607093] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=17633, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:57.607143] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2250496}], scn_array=[cnt:1, {val:1736326317454295000, v:0}], curr_size=122, accum_size=366, time_cost=26472) [2025-01-08 00:51:57.607307] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=150] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=1825, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:57.607400] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=45, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:57.610667] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58600][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D4-0-0] [lt=29] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326317619466, keepalive_ts:-1}) [2025-01-08 00:51:57.610701] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58600][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D4-0-0] [lt=33] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326317619466, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:57.611017] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58633][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FE-0-0] [lt=14] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326317624429, keepalive_ts:-1}) [2025-01-08 00:51:57.611048] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58633][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FE-0-0] [lt=30] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326317624429, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:57.612096] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58646][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B6B-0-0] [lt=37][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18386, log_task={header:{begin_lsn:{lsn:2237844}, end_lsn:{lsn:2237966}, log_id:18386, min_scn:{val:1736326317048856003, v:0}, max_scn:{val:1736326317048856003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2237722}, prev_proposal_id:1, committed_end_lsn:{lsn:2237844}, data_checksum:940009244, accum_checksum:2500413237, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326317500526, freeze_ts:1736326317500526, submit_ts:1736326317500533, flushed_ts:1736326317590951, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:90418}, fs_cb_begin_ts=1736326317612069, log_life_time=111543) [2025-01-08 00:51:57.612325] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=19838, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:57.612343] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43612382}], scn_array=[cnt:1, {val:1736326317364940001, v:0}], curr_size=122, accum_size=1300, time_cost=114808) [2025-01-08 00:51:57.612370] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=1241, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:57.612528] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=85, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:57.612565] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=11, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:57.612575] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=68001, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:57.612582] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=33276, log_task={header:{begin_lsn:{lsn:43612382}, end_lsn:{lsn:43612504}, log_id:33276, min_scn:{val:1736326317364940001, v:0}, max_scn:{val:1736326317364940001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43612382}, data_checksum:667309872, accum_checksum:2504434278, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326317496306, freeze_ts:1736326317496321, submit_ts:1736326317496376, flushed_ts:1736326317612401, gen_to_freeze cost time:15, gen_to_submit cost time:70, submit_to_flush cost time:116025}, fs_cb_begin_ts=1736326317612558, log_life_time=116252) [2025-01-08 00:51:57.612631] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=110, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:57.613471] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9687-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:51:57.615590] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=8950, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:57.615714] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18737979}], scn_array=[cnt:1, {val:1736326317157901003, v:0}], curr_size=122, accum_size=2066, time_cost=15197) [2025-01-08 00:51:57.615743] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2066) [2025-01-08 00:51:57.615792] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=43] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=254, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:57.615810] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=18] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:57.615932] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=84][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84477760}], scn_array=[cnt:1, {val:1736326317192724002, v:0}], curr_size=122, accum_size=244, time_cost=16502) [2025-01-08 00:51:57.616294] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2216616}], scn_array=[cnt:1, {val:1736326317157901003, v:0}], curr_size=122, accum_size=366, time_cost=15575) [2025-01-08 00:51:57.616453] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=82, avg_log_submit_to_flush_time=21789, avg_log_submit_to_slide_time=27066) [2025-01-08 00:51:57.616575] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2066, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:57.617198] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:57.617706] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=1] kill and revert session(conn.sessid_=3221587631, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.617737] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] sock ref clean, do destroy(*s={this:0x2ad8efffc030, session_id:3221587631, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.617775] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] unlock session of tenant(conn.sessid_=3221587631, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.617786] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221587631, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CE9-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:123, rec_:123, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:1, seq_:3}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:4}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:5}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.617862] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221587631) [2025-01-08 00:51:57.617883] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] mark session id unused(sessid=3221587631) [2025-01-08 00:51:57.629566] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2237966}], scn_array=[cnt:1, {val:1736326317454295001, v:0}], curr_size=122, accum_size=366, time_cost=14086) [2025-01-08 00:51:57.635781] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43612504}], scn_array=[cnt:1, {val:1736326317454295000, v:0}], curr_size=1121, accum_size=2421, time_cost=23058) [2025-01-08 00:51:57.657346] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=86] current monitor number(seq_=-1) [2025-01-08 00:51:57.659306] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1736326317659303, bytes: 884468329, bw: 0.246728 MB/s, add_ts: 1001421, add_bytes: 259081 [2025-01-08 00:51:57.672531] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] clock out of order(cur_ts=1736326317672528, cur_ts_=1736326317629447, delta=43081) [2025-01-08 00:51:57.675462] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:51:57.675560] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:57.676354] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:57.676473] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=2] kill and revert session(conn.sessid_=3221587635, proxy_sessid=0, ret=0) [2025-01-08 00:51:57.676491] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] sock ref clean, do destroy(*s={this:0x2ad8efe0c030, session_id:3221587635, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:206}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:51:57.676595] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=72] unlock session of tenant(conn.sessid_=3221587635, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:51:57.676622] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] connection close(sessid=3221587635, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CED-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:119, rec_:119, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:2, seq_:4}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:5}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:51:57.676807] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=8] free session successfully(sessid=3221587635) [2025-01-08 00:51:57.676823] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=15] mark session id unused(sessid=3221587635) [2025-01-08 00:51:57.684664] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48CD-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:57.684707] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48CD-0-0] [lt=42] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=68802, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:57.684721] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48CD-0-0] [lt=11][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18490, log_task={header:{begin_lsn:{lsn:2250496}, end_lsn:{lsn:2250618}, log_id:18490, min_scn:{val:1736326317454295000, v:0}, max_scn:{val:1736326317454295000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2250374}, prev_proposal_id:1, committed_end_lsn:{lsn:2250496}, data_checksum:1715074492, accum_checksum:1315866312, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326317579620, freeze_ts:1736326317579620, submit_ts:1736326317579775, flushed_ts:1736326317607343, gen_to_freeze cost time:0, gen_to_submit cost time:155, submit_to_flush cost time:27568}, fs_cb_begin_ts=1736326317684649, log_life_time=105029) [2025-01-08 00:51:57.684806] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF6F-0-0] [lt=33] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326317794893, keepalive_ts:-1}) [2025-01-08 00:51:57.684825] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF6F-0-0] [lt=18] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326317794893, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:57.687784] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D52-0-0] [lt=36] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:57.689185] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDB-0-0] [lt=17] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:57.689307] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDB-0-0] [lt=118] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:57.689507] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:57.692849] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:57.698010] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:57.698415] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=2449, avg_group_log_size=272) [2025-01-08 00:51:57.698933] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] no need to update min/max version(new_version={val:1736326315849992001, v:0}, min_version={val:1736326315849992001, v:0}, max_version={val:1736326315849992001, v:0}) [2025-01-08 00:51:57.699560] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF35-0-0] [lt=20] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326317719544, keepalive_ts:-1}) [2025-01-08 00:51:57.699584] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF35-0-0] [lt=22] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326317719544, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:57.699998] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2250618}], scn_array=[cnt:1, {val:1736326317454295001, v:0}], curr_size=122, accum_size=488, time_cost=12354) [2025-01-08 00:51:57.705751] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:57.705811] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:57.705898] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=39, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:57.705943] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:57.705962] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:57.705977] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:57.707009] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B6D-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:57.707145] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B6D-0-0] [lt=122] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=8, avg_log_batch_cnt=0, total_group_log_size=976, avg_group_log_size=122) [2025-01-08 00:51:57.707166] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B6D-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=976) [2025-01-08 00:51:57.707250] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B6D-0-0] [lt=70] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=15, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:57.707260] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B6D-0-0] [lt=10] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=96782, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:57.707268] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B6D-0-0] [lt=6] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=43818, avg_log_submit_to_slide_time=96761) [2025-01-08 00:51:57.707664] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:57.708742] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=976, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:57.708780] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18297, miss_cnt:0, cache_read_size:2238088, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:57.709021] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085177-0-0] [lt=13] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:57.720902] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43613625}], scn_array=[cnt:1, {val:1736326317629447000, v:0}], curr_size=122, accum_size=2543, time_cost=11928) [2025-01-08 00:51:57.730964] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6ED-0-0] [lt=5][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193822644796, cache_obj->added_lc()=false, cache_obj->get_object_id()=13693, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:57.749921] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=32] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:57.749951] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=29] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:57.759239] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84993-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:57.759310] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84993-0-0] [lt=52] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:57.774734] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46500][T1_L0_G0][T1][YB42C0A89F8B-00062B2D255BAD4A-0-0] [lt=36] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.760760459899902344e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=880, Throttle Trigger(MB)=528, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.962595039795435881e+01) [2025-01-08 00:51:57.775020] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46500][T1_L0_G0][T1][YB42C0A89F8B-00062B2D255BAD4A-0-0] [lt=98] get number(ret=0, service_type_=0, range=1, base_id=1736326317771747000, start_id=1736326317771747000, end_id=1736326317771747001) [2025-01-08 00:51:57.776525] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:57.777697] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:57.781491] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:51:57.781553] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:57.784896] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC3-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:57.787195] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC3-0-0] [lt=20][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:57.788677] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=68] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:51:57.803687] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6720679936, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:57.805779] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:57.806872] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] no need to update min/max version(new_version={val:1736326315849992001, v:0}, min_version={val:1736326315849992001, v:0}, max_version={val:1736326315849992001, v:0}) [2025-01-08 00:51:57.809762] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:51:57.809804] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] try gc retain ctx [2025-01-08 00:51:57.809829] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] try to retry start_working [2025-01-08 00:51:57.809861] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] finish update min start scn(max_decided_scn={val:1736326317157901003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326307358778003, v:0}, keep_alive_scn_:{val:1736326312645069001, v:0}, update_ts_:1736326317803522}) [2025-01-08 00:51:57.810897] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326330087570000, latest_log_ts:{val:1736326313644270001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:57.811135] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=111] finish update min start scn(max_decided_scn={val:1736326317157901003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326307358778003, v:0}, keep_alive_scn_:{val:1736326312645069001, v:0}, update_ts_:1736326317803522}) [2025-01-08 00:51:57.812122] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=35] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326312645069002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:57.812328] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=97] finish update min start scn(max_decided_scn={val:1736326317454295001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326310862856000, v:0}, keep_alive_scn_:{val:1736326310862856000, v:0}, update_ts_:1736326317803522}) [2025-01-08 00:51:57.813335] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326312811284000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:57.813440] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=96] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:51:57.813466] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:2238088}, scn={val:1736326317454295002, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326317707245, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2238088}, next_to_submit_scn_:{val:1736326317454295002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2237966}, end_lsn:{lsn:2238088}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2238210}, readable_log_tail:{lsn:2238210}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64866558, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326317454295001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326317454295001, v:0}, data_checksum:2124447694, flag:18042}}, init_mode_version:0, accumulate_checksum:1511697140, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326317454295001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18297, miss_cnt:0, cache_read_size:2238088, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:57.813565] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=96] finish update min start scn(max_decided_scn={val:1736326317454295001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326309349119001, v:0}, keep_alive_scn_:{val:1736326314165163003, v:0}, update_ts_:1736326317803522}) [2025-01-08 00:51:57.814607] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326312813632000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:58.128203] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4016] task executed too delay(delay_time=318587, *p_task_={this:0x2ad8efe05570, caller:ob_leader_coordinator.cpp:mtl_start:104, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8efe05850, time_wheel:0x2ad8cf1f7990, thread_pool:0x2ad8cf1f7170, is_running:0, time_interval:5.00s, expected_run_time:00:52:02.799, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:58.134335] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:58.134717] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.134777] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.134824] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.134870] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.139049] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46102][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=20] expand thread count(this=0x5572d527f900, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:58.140175] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:58.143677] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=35] gc stale ls task succ [2025-01-08 00:51:58.143747] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=284] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1}, {id:1001}, {id:1003}]) [2025-01-08 00:51:58.143859] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:58.143891] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-01-08 00:51:58.143905] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:58.143924] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:58.143946] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193821785167, flush_io_finished_ret_:0, flush_io_finished_round_:3804, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:58.151728] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:58.151808] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=77][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:58.154512] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87187-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:58.154596] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:58.154611] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:58.154838] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:58.154981] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:51:58.157307] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185450-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:58.159167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:58.160672] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46508][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D2-0-0] [lt=87][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:58.160756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:58.161139] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [50419][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=26] expand thread count(this=0x2ad8c0eb0e70, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:58.163972] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=0] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:58.164087] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=179] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:5, threads_idle_time:15264976321, tenant_id:500}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:58.167395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50526][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=69][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:58.167710] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:58.167779] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46308][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=20] expand thread count(this=0x2ad8881fc3b0, max_thread_cnt=128, cur_thread_count=4, inc_cnt=1, queue_size=4) [2025-01-08 00:51:58.167810] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] ====== check clog disk timer task ====== [2025-01-08 00:51:58.167829] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:51:58.167968] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=108] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:51:58.167997] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=20] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=90) [2025-01-08 00:51:58.171002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=70][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:51:58.171401] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [46106][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=137][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:9246, tid:46491}]) [2025-01-08 00:51:58.171838] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:51:58.172418] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6705999872, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:58.178291] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:58.178765] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326318178738], gts:1736326317803522000, latest_srr:[mts=1736326318178738]}) [2025-01-08 00:51:58.178832] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=45] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326318178811], gts:1736326317803522001, latest_srr:[mts=1736326318178811]}) [2025-01-08 00:51:58.179060] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326318179049], gts:1736326317803522000, latest_srr:[mts=1736326318179049]}) [2025-01-08 00:51:58.179738] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58600][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=15] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:58.180957] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4015] clock out of order(cur_ts=1736326318180955, cur_ts_=1736326317803522, delta=377433) [2025-01-08 00:51:58.181314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46274][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=113][errcode=0] task scheduled too delay(scheduled_delay=156288, expected_run_ts_us_=1736326318025022, ObClockGenerator::getRealClock()=1736326318181312, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:58.250, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:58.181449] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:58.181494] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:58.182320] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=61544, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:58.182361] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84478004}], scn_array=[cnt:1, {val:1736326317771747000, v:0}], curr_size=545, accum_size=911, time_cost=407019) [2025-01-08 00:51:58.182528] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=156] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=3128, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:58.182650] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=88, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:58.182696] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=50, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:58.183361] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50380][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=0] task scheduled too delay(scheduled_delay=375695, expected_run_ts_us_=1736326317807663, ObClockGenerator::getRealClock()=1736326318183359, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:1, time_interval:100.00ms, expected_run_time:00:51:57.807, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:58.183724] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46141][pnio1][T0][YB42C0A89F89-00062B2D21F89E2F-0-0] [lt=29][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5387, fly_ts=356646, send_timestamp=1736326317827077, peer="192.168.159.139:45058", sz=480) [2025-01-08 00:51:58.186296] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58631][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D71-0-0] [lt=12][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:84475433}, new_end_lsn:{lsn:84475677}, old_advance_time_us:1736326316578562, new_ack_time_us:1736326318186291, advance delay(us):1607729}) [2025-01-08 00:51:58.187179] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46142][pnio1][T0][YB42C0A89F89-00062B2D18189E6B-0-0] [lt=24][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5387, fly_ts=359143, send_timestamp=1736326317828034, peer="192.168.159.139:45062", sz=479) [2025-01-08 00:51:58.192758] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE955-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1736326317803522000, start_id=1736326317803522001, end_id=1736326317803522002) [2025-01-08 00:51:58.196568] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C2BB638-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1736326317803522000, start_id=1736326317803522003, end_id=1736326317803522004) [2025-01-08 00:51:58.197431] INFO [LIB] stat (utility.h:1210) [58633][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A12-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:58.197468] INFO [LIB] stat (utility.h:1210) [58633][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A12-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=95278, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:58.197487] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326317771747000, v:0}, log_ts:{val:1736326317771747000, v:0}, lsn:{lsn:84478060}, submit_ts:1736326317775090}, this:0x2ad89c086358, is_inited_:true, trans_id:{txid:28251}, ls_id:{id:1}, ctx:0x2ad89c082950, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=422393, ctx_lock_wait_time=1) [2025-01-08 00:51:58.197481] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58633][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A12-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=45104, log_task={header:{begin_lsn:{lsn:84478004}, end_lsn:{lsn:84478549}, log_id:45104, min_scn:{val:1736326317771747000, v:0}, max_scn:{val:1736326317771747000, v:0}, data_len:489, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84478004}, data_checksum:272801103, accum_checksum:1581913116, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326317775100, freeze_ts:1736326317775109, submit_ts:1736326317775180, flushed_ts:1736326318182570, gen_to_freeze cost time:9, gen_to_submit cost time:80, submit_to_flush cost time:407390}, fs_cb_begin_ts=1736326318197402, log_life_time=422302) [2025-01-08 00:51:58.198196] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=89] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:58.198348] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:51:58.199634] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2250740}], scn_array=[cnt:1, {val:1736326317454295002, v:0}], curr_size=122, accum_size=610, time_cost=415118) [2025-01-08 00:51:58.200080] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46144][pnio1][T0][YB42C0A89F8B-00062B2D181C5D70-0-0] [lt=150][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5378, fly_ts=370461, send_timestamp=1736326317829617, peer="192.168.159.139:45074", sz=263) [2025-01-08 00:51:58.202433] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Slow write(duration=40434, write_len=4096) [2025-01-08 00:51:58.202452] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=18] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8826, offset=2507544}, end_cursor:ObLogCursor{file_id=1, log_id=8827, offset=2507849}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ce4050, len:3304, log_data_len:305, seq:8826, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:58.206798] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D70-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2286) [2025-01-08 00:51:58.208367] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [46143][pnio1][T0][YB42C0A89F8B-00062B2D21BC3B6F-0-0] [lt=55][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5377, fly_ts=381535, send_timestamp=1736326317826830, peer="192.168.159.139:45082", sz=418) [2025-01-08 00:51:58.211617] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B6F-0-0] [lt=25][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18388, log_task={header:{begin_lsn:{lsn:2238088}, end_lsn:{lsn:2238210}, log_id:18388, min_scn:{val:1736326317454295002, v:0}, max_scn:{val:1736326317454295002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2237966}, prev_proposal_id:1, committed_end_lsn:{lsn:2238088}, data_checksum:940009244, accum_checksum:3426309439, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326317707003, freeze_ts:1736326317707003, submit_ts:1736326317707134, flushed_ts:1736326317713638, gen_to_freeze cost time:0, gen_to_submit cost time:131, submit_to_flush cost time:6504}, fs_cb_begin_ts=1736326318211597, log_life_time=504594) [2025-01-08 00:51:58.212447] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] no need to update min/max version(new_version={val:1736326315849992001, v:0}, min_version={val:1736326315849992001, v:0}, max_version={val:1736326315849992001, v:0}) [2025-01-08 00:51:58.213465] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2216860}], scn_array=[cnt:1, {val:1736326317629447001, v:0}], curr_size=113, accum_size=601, time_cost=398354) [2025-01-08 00:51:58.213537] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=18214, log_task={header:{begin_lsn:{lsn:2216860}, end_lsn:{lsn:2216973}, log_id:18214, min_scn:{val:1736326317629447001, v:0}, max_scn:{val:1736326317629447001, v:0}, data_len:57, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2216860}, data_checksum:1564946031, accum_checksum:2742856578, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326317812226, freeze_ts:1736326317812242, submit_ts:1736326317812302, flushed_ts:1736326318213514, gen_to_freeze cost time:16, gen_to_submit cost time:76, submit_to_flush cost time:401212}, fs_cb_begin_ts=1736326318213527, log_life_time=401301) [2025-01-08 00:51:58.213736] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18738223}], scn_array=[cnt:1, {val:1736326317629447001, v:0}], curr_size=113, accum_size=235, time_cost=398657) [2025-01-08 00:51:58.213791] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=110][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=19854, log_task={header:{begin_lsn:{lsn:18738223}, end_lsn:{lsn:18738336}, log_id:19854, min_scn:{val:1736326317629447001, v:0}, max_scn:{val:1736326317629447001, v:0}, data_len:57, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18738223}, data_checksum:2031283194, accum_checksum:3019373474, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326317811025, freeze_ts:1736326317811030, submit_ts:1736326317811101, flushed_ts:1736326318213776, gen_to_freeze cost time:5, gen_to_submit cost time:76, submit_to_flush cost time:402675}, fs_cb_begin_ts=1736326318213785, log_life_time=402760) [2025-01-08 00:51:58.214676] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326308184989, end_time=1736326318214673, dag_status={}) [2025-01-08 00:51:58.214722] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=43] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:58.214736] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:58.214743] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:58.214750] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:58.214757] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:58.214764] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:58.214771] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:58.214778] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:58.214784] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:58.214792] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214803] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214811] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214819] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214827] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214835] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214843] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214865] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214874] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214882] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214889] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214898] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214919] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.214999] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=79] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215087] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=85] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215103] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215114] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215125] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215139] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215152] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215165] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215209] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=44] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215490] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=278] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215553] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215573] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215588] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215602] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215615] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215751] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=135] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215812] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215831] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215846] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215861] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215874] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215885] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215899] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215953] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=52] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215987] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=34] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.215997] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216005] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216014] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216022] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216030] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216038] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216046] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216054] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216061] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216069] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216085] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216092] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216100] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216108] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216116] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216124] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216132] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216139] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216147] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216155] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216163] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216171] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216179] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216186] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216194] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216289] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=93] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216303] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216312] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216321] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216329] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216337] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216344] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216353] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216361] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:58.216370] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:58.216378] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:58.216387] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:58.216392] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:58.216397] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:58.216402] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:58.216408] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:58.216413] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:58.216418] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:58.216423] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:58.216430] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:58.216788] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46390][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D3-0-0] [lt=92][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:58.218866] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84478549}], scn_array=[cnt:1, {val:1736326317771747001, v:0}], curr_size=122, accum_size=1033, time_cost=35311) [2025-01-08 00:51:58.219121] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=75] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=1869, avg_log_gen_to_submit_time=1982, avg_log_submit_to_flush_time=68432, avg_log_submit_to_slide_time=87811) [2025-01-08 00:51:58.219187] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=119] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1863, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:58.221144] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E77-0-0] [lt=20] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:58.221241] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E77-0-0] [lt=19] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326317629447000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326317629447000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:58.224822] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=110] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1512, avg_group_log_size=168) [2025-01-08 00:51:58.224850] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1512) [2025-01-08 00:51:58.231460] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E77-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=8622) [2025-01-08 00:51:58.235576] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852BC-0-0] [lt=40] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:51:58.237409] WDIAG get_refactored (ob_hashtable.h:1120) [46500][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D5-0-0] [lt=21][errcode=-4201] hashtable not init [2025-01-08 00:51:58.237798] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D0C8-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:58.237956] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58585][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48D1-0-0] [lt=16][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18492, log_task={header:{begin_lsn:{lsn:2250740}, end_lsn:{lsn:2250862}, log_id:18492, min_scn:{val:1736326317454295002, v:0}, max_scn:{val:1736326317454295002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2250618}, prev_proposal_id:1, committed_end_lsn:{lsn:2250740}, data_checksum:1715074492, accum_checksum:1324254204, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326317784314, freeze_ts:1736326317784314, submit_ts:1736326317784327, flushed_ts:1736326318199691, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:415364}, fs_cb_begin_ts=1736326318237943, log_life_time=453629) [2025-01-08 00:51:58.240633] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CA-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:51:58.240676] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CA-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326317454295002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326317454295002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:58.243617] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:5, threads_idle_time:7995583490, tenant_id:1002}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:58.243663] WDIAG [STORAGE.TRANS] is_external_consistent (ob_ts_mgr.cpp:1064) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4002] invalid argument(ret=-4002, ret="OB_INVALID_ARGUMENT", tenant_id=18446744073709551615) [2025-01-08 00:51:58.245583] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=138] ====== checkpoint timer task ====== [2025-01-08 00:51:58.246420] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:58.246460] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=40] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:58.246530] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326314647131003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326330087570000, latest_log_ts:{val:1736326313644270001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:58.246607] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=74] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:51:58.246626] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:58.246639] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326313644270001, v:0}) [2025-01-08 00:51:58.246647] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:51:58.246661] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:58.246679] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:58.246691] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:58.246724] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:58.246754] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:58.247565] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:58.247604] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=38] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:58.247648] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326312645069002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:58.247712] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=62] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:51:58.247726] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:51:58.247739] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:58.247754] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:51:58.247770] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2217095}, scn={val:1736326317629447000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:58.251555] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [58656][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=0] ====== traversal_flush timer task ====== [2025-01-08 00:51:58.251697] INFO [MDS] flush (mds_table_mgr.cpp:195) [58656][T1_TimerWK4][T1][YB42C0A89F89-00062B2D2898458E-0-0] [lt=37] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326317771747001, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329254996000, latest_log_ts:{val:1736326312592647001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:58.251803] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [58656][T1_TimerWK4][T1][YB42C0A89F89-00062B2D2898458E-0-0] [lt=100] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:51:58.251832] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=43] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:15265148619, tenant_id:500}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:58.252795] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:7995583643, tenant_id:1002}, cur_thread_count=5, cnt=1, new_thread_count=6) [2025-01-08 00:51:58.255003] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=106] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:51:58.255072] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46308][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=15] expand thread count(this=0x2ad8881fc3b0, max_thread_cnt=128, cur_thread_count=5, inc_cnt=1, queue_size=5) [2025-01-08 00:51:58.255793] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8827, offset=2507849}, end_cursor:ObLogCursor{file_id=1, log_id=8828, offset=2508146}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ce0050, len:2999, log_data_len:297, seq:8827, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:58.255865] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326317629447000, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:58.256052] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=177][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:58.256076] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:51:58.257313] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:58.257422] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=109] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:58.257673] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=225] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326312811284000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:58.257743] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=67] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:51:58.257759] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:51:58.257995] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=226] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:58.258017] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:51:58.258033] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2250984}, scn={val:1736326317803522002, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:58.258486] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:6, threads_idle_time:15193854797, tenant_id:1}, cur_thread_count=6, cnt=1, new_thread_count=7) [2025-01-08 00:51:58.262588] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:58.264760] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50625][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC42E-0-0] [lt=42][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:58.266925] INFO [PALF] pread (log_reader.cpp:130) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=5915) [2025-01-08 00:51:58.270838] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:58.270896] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:7, bw:28086, [delay/us]:prepare:7, schedule:823, submit:3403, rt:52360, total:56594) [2025-01-08 00:51:58.270913] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=1, iops=1, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:58.270971] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:7, avg_bw:28088, [delay/us]: prepare:7, schedule:823, submit:3403, device:52360, total:56594) [2025-01-08 00:51:58.270985] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:58.271025] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:12288 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:7, bw:31210, [delay/us]:prepare:9, schedule:1220, submit:2092, rt:16240, total:19562) [2025-01-08 00:51:58.271035] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1001, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=1, iops=1, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:58.271071] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:7, avg_bw:31212, [delay/us]: prepare:9, schedule:1220, submit:2092, device:16240, total:19562) [2025-01-08 00:51:58.271081] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:58.271103] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:2, bw:6242, [delay/us]:prepare:5, schedule:457, submit:1, rt:22728, total:23192) [2025-01-08 00:51:58.271125] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4206, iops:28, bw:118611, [delay/us]:prepare:5, schedule:20480, submit:4455, rt:21322, total:46263) [2025-01-08 00:51:58.271238] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=107] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:2, bw:6242, [delay/us]:prepare:3, schedule:480, submit:466, rt:4004, total:4954) [2025-01-08 00:51:58.271273] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS TENANT](tenant_id=1002, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=7, iops=7, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:58.271337] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:2, avg_bw:6242, [delay/us]: prepare:5, schedule:457, submit:1, device:22728, total:23192) [2025-01-08 00:51:58.271354] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:2, avg_bw:6242, [delay/us]: prepare:3, schedule:480, submit:466, device:4004, total:4954) [2025-01-08 00:51:58.271362] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4207, avg_iops:29, avg_bw:118608, [delay/us]: prepare:5, schedule:20480, submit:4455, device:21322, total:46263) [2025-01-08 00:51:58.271373] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:58.271395] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3121, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:970, total:980) [2025-01-08 00:51:58.271420] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:58.271459] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3121, [delay/us]: prepare:0, schedule:0, submit:600000000, device:970, total:980) [2025-01-08 00:51:58.271479] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:58.271495] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:58.271503] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:58.271511] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:58.271519] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=191kB/s, net_out=172kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:58.274466] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB620-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:51:58.274509] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB620-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326317454295003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326317454295003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:58.274810] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8828, offset=2508146}, end_cursor:ObLogCursor{file_id=1, log_id=8829, offset=2508443}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894ce8050, len:2702, log_data_len:297, seq:8828, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:58.274913] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326317803522002, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:58.274976] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:58.275002] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:51:58.276268] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:58.276322] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=54] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:58.276375] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326312813632000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326313644270003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:58.276512] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=133] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:51:58.276531] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:51:58.276551] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:58.276569] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:51:58.276585] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2238332}, scn={val:1736326317454295003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:51:58.278010] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:58.278147] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:58.282884] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:58.283121] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58657][T1002_TimerWK5][T1002][Y0-0000000000000000-0-0] [lt=0] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:58.283214] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=94] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:15193871531, tenant_id:1}, cur_thread_count=6, cnt=1, new_thread_count=7) [2025-01-08 00:51:58.292986] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=41] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:58.293667] INFO [PALF] pread (log_reader.cpp:130) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=8858) [2025-01-08 00:51:58.297662] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=26] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8829, offset=2508443}, end_cursor:ObLogCursor{file_id=1, log_id=8830, offset=2508740}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cec050, len:2405, log_data_len:297, seq:8829, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:58.297738] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326317454295003, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:51:58.297776] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:58.297802] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:51:58.297813] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:51:58.298155] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1736326318298151, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000444, add_bytes: 0 [2025-01-08 00:51:58.303934] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:58.308589] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58633][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=90] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:51:58.318737] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get number(ret=0, service_type_=0, range=1, base_id=1736326318316526000, start_id=1736326318316526000, end_id=1736326318316526001) [2025-01-08 00:51:58.318992] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2408) [2025-01-08 00:51:58.319107] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:149) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C2CA-0-0] [lt=21] transaction rpc statistics(total_rt=3540, total_process=14, avg_rt=236) [2025-01-08 00:51:58.321132] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.321185] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.327728] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:192K/47:15416354 disk:192K/47:15416354 t1m2:24K/6:112563 t1m2g0:24K/6:112563 t1002m2:136K/34:20757402 t1002m2g0:136K/34:20757402 t1001m2:32K/7:2588303 t1001m2g0:32K/7:2588303 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:58.338864] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48D3-0-0] [lt=43] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=854, avg_group_log_size=122) [2025-01-08 00:51:58.338922] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48D3-0-0] [lt=54] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=854) [2025-01-08 00:51:58.341141] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5315852366, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:58.341340] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=36] recycle one thread(this=0x2ad8885abcf0, total=4, remain=3) [2025-01-08 00:51:58.341358] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5315852366, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:51:58.341644] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84478671}], scn_array=[cnt:1, {val:1736326318316526000, v:0}], curr_size=586, accum_size=1619, time_cost=20602) [2025-01-08 00:51:58.342118] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18738458}], scn_array=[cnt:1, {val:1736326317803522001, v:0}], curr_size=122, accum_size=479, time_cost=14640) [2025-01-08 00:51:58.342459] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326316161938000, v:0}, new_max_version={val:1736326316161938000, v:0}, last_min_version={val:1736326315849992001, v:0}, last_max_version={val:1736326315849992001, v:0}, total_time_us=25843, delta_us=2180519) [2025-01-08 00:51:58.342489] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326316161938000, v:0}, min_version={val:1736326316161938000, v:0}, max_version={val:1736326316161938000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:58.347299] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=854) [2025-01-08 00:51:58.352436] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50625][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB620-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1736326318349518000, start_id=1736326318349518000, end_id=1736326318349518001) [2025-01-08 00:51:58.352665] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2217095}], scn_array=[cnt:1, {val:1736326317803522001, v:0}], curr_size=122, accum_size=845, time_cost=25141) [2025-01-08 00:51:58.352701] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=845) [2025-01-08 00:51:58.353310] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A5C-0-0] [lt=134] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=845) [2025-01-08 00:51:58.359893] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884EFF-0-0] [lt=44] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:51:58.372565] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326314171314, last_first_register_time_us=1736324454612308, register_time_us=1736326314171314, curr_time_us=1736326318372561) [2025-01-08 00:51:58.372697] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=40] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326318372607, last_submit_register_req_time_us=1736326318372607, register_parent_reason=1) [2025-01-08 00:51:58.372717] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:51:58.373079] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46381][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6EF-0-0] [lt=51][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193823286911, cache_obj->added_lc()=false, cache_obj->get_object_id()=13695, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:58.375419] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58655][TimerWK5][T0][Y0-0000000000000000-0-0] [lt=0] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6718582784, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:58.375578] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=28] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:51:58.375729] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=32] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:51:58.375975] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2238454}], scn_array=[cnt:1, {val:1736326317803522002, v:0}], curr_size=122, accum_size=854, time_cost=11250) [2025-01-08 00:51:58.376026] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=854) [2025-01-08 00:51:58.376488] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=78][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=631246848, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:51:58.376827] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=335] === Run print tenant memory usage task === [2025-01-08 00:51:58.376876] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=16] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,874,944 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 633,344,000 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,318,375,966 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 158,284,580 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,063,759,872 max_mem_memstore_can_get_now= 791,422,936 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,318,375,966 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 70,704,720 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 651,907,072 max_mem_memstore_can_get_now= 353,523,698 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,318,375,966 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 240,253,320 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,118,584,832 max_mem_memstore_can_get_now= 1,201,266,668 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:51:58.377173] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=10] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,657,314,304 total_hold= 3,856,662,528 used= 3,498,741,760 freelists_hold= 158,572,544 total_maps= 1,637 total_unmaps= 245 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,874,944 virtual_memory_used= 4,829,478,912 [CHUNK_MGR] 2 MB_CACHE: hold= 56,623,104 count= 27 pushes= 1,844,586 pops= 1,844,559 maps= 1,506 unmaps= 240 [CHUNK_MGR] 4 MB_CACHE: hold= 7,745,536 count= 3 pushes= 6,154 pops= 6,151 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,156 pops= 3,152 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:51:58.377282] INFO print (ob_malloc_time_monitor.cpp:37) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=90] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 10220, delta_count= 2, avg_cost_time= 5110 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 24626, delta_count= 1, avg_cost_time= 24626 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-01-08 00:51:58.377507] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:51:58.379564] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58660][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:58.382332] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58661][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:58.382389] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:51:58.385667] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:58.387986] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43613747}], scn_array=[cnt:1, {val:1736326317793567000, v:0}], curr_size=589, accum_size=3132, time_cost=580496) [2025-01-08 00:51:58.388017] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3132) [2025-01-08 00:51:58.389503] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3678) [2025-01-08 00:51:58.389653] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=10556, avg_group_log_size=812) [2025-01-08 00:51:58.389702] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=8190, avg_log_gen_to_submit_time=8256, avg_log_submit_to_flush_time=126561, avg_log_submit_to_slide_time=128185) [2025-01-08 00:51:58.389716] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=33279, log_task={header:{begin_lsn:{lsn:43613747}, end_lsn:{lsn:43614336}, log_id:33279, min_scn:{val:1736326317793567000, v:0}, max_scn:{val:1736326317793567000, v:0}, data_len:533, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43613747}, data_checksum:1332607544, accum_checksum:781843307, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326317802901, freeze_ts:1736326317802909, submit_ts:1736326317802983, flushed_ts:1736326318389488, gen_to_freeze cost time:8, gen_to_submit cost time:82, submit_to_flush cost time:586505}, fs_cb_begin_ts=1736326318389678, log_life_time=586777) [2025-01-08 00:51:58.393753] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=82][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326317793567000, v:0}, log_ts:{val:1736326317793567000, v:0}, lsn:{lsn:43613803}, submit_ts:1736326317802890}, this:0x2ad8fb7086d8, is_inited_:true, trans_id:{txid:67341}, ls_id:{id:1}, ctx:0x2ad8fb704cd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=590858, ctx_lock_wait_time=1) [2025-01-08 00:51:58.404108] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43614336}], scn_array=[cnt:1, {val:1736326317803522000, v:0}], curr_size=3612, accum_size=3612, time_cost=14212) [2025-01-08 00:51:58.404703] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326317803522000, v:0}, log_ts:{val:1736326317803522000, v:0}, lsn:{lsn:43614392}, submit_ts:1736326317807410}, this:0x2ad8fb6c4ed8, is_inited_:true, trans_id:{txid:67342}, ls_id:{id:1}, ctx:0x2ad8fb6c14d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=597289, ctx_lock_wait_time=1) [2025-01-08 00:51:58.404866] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=42] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:51:58.404911] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326317803522003, v:0}, log_ts:{val:1736326317803522003, v:0}, lsn:{lsn:43614991}, submit_ts:1736326318196706}, this:0x2ad8fb69ef58, is_inited_:true, trans_id:{txid:67343}, ls_id:{id:1}, ctx:0x2ad8fb69b550, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=208204, ctx_lock_wait_time=0) [2025-01-08 00:51:58.404982] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326318229695000, v:0}, log_ts:{val:1736326318229695000, v:0}, lsn:{lsn:43615481}, submit_ts:1736326318231446}, this:0x2ad8fb6a32d8, is_inited_:true, trans_id:{txid:67344}, ls_id:{id:1}, ctx:0x2ad8fb69f8d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=173534, ctx_lock_wait_time=0) [2025-01-08 00:51:58.411348] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8830, offset=2508740}, end_cursor:ObLogCursor{file_id=1, log_id=8831, offset=2509037}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cf0050, len:2108, log_data_len:297, seq:8830, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:58.411771] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E77-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:51:58.411873] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E77-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326317803522001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326317803522001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:58.422409] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46384][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6F4-0-0] [lt=51] get number(ret=0, service_type_=0, range=1, base_id=1736326318418972000, start_id=1736326318418972000, end_id=1736326318418972001) [2025-01-08 00:51:58.426610] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2184) [2025-01-08 00:51:58.426855] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=50] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:51:58.444436] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DAED-0-0] [lt=47] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=41, avg_log_submit_to_flush_time=62337, avg_log_submit_to_slide_time=100660) [2025-01-08 00:51:58.446548] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=113] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=976, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:58.446666] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=106] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18402, miss_cnt:0, cache_read_size:2251228, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:58.448860] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=95] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8831, offset=2509037}, end_cursor:ObLogCursor{file_id=1, log_id=8832, offset=2509334}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cf8050, len:1811, log_data_len:297, seq:8831, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:58.451389] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8832, offset=2509334}, end_cursor:ObLogCursor{file_id=1, log_id=8833, offset=2509631}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cf4050, len:1514, log_data_len:297, seq:8832, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:58.458567] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.460345] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.478319] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B75-0-0] [lt=46][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18391, log_task={header:{begin_lsn:{lsn:2238454}, end_lsn:{lsn:2238576}, log_id:18391, min_scn:{val:1736326317803522002, v:0}, max_scn:{val:1736326317803522002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2238332}, prev_proposal_id:1, committed_end_lsn:{lsn:2238454}, data_checksum:940009244, accum_checksum:1359497026, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326318362376, freeze_ts:1736326318362376, submit_ts:1736326318362389, flushed_ts:1736326318376063, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:13674}, fs_cb_begin_ts=1736326318478291, log_life_time=115915) [2025-01-08 00:51:58.486460] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:51:58.486516] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:58.503773] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2238576}], scn_array=[cnt:1, {val:1736326317803522003, v:0}], curr_size=122, accum_size=122, time_cost=25325) [2025-01-08 00:51:58.508537] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get number(ret=0, service_type_=0, range=1, base_id=1736326318503739000, start_id=1736326318503739000, end_id=1736326318503739001) [2025-01-08 00:51:58.522966] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=130] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.541507] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15266697925, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:58.541694] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=180][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:58.541713] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] try_wait return eagain(this=0x5572d527f900, thread=0x2ad8bf5d5af0) [2025-01-08 00:51:58.541723] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15266697925, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:58.541785] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=61] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15195420869, tenant_id:1}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:58.541802] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:58.541810] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try_wait return eagain(this=0x2ad8881fc3b0, thread=0x2ad8d9dfd410) [2025-01-08 00:51:58.541816] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15195420869, tenant_id:1}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:58.541825] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7997140439, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:58.541884] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:58.541892] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try_wait return eagain(this=0x2ad8c0eb0e70, thread=0x2ad8f21ff700) [2025-01-08 00:51:58.541899] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7997140439, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:58.544292] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.588616] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:58.589714] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-4016] task executed too delay(delay_time=58404, *p_task_={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:0, time_interval:250.00ms, expected_run_time:00:51:58.775, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:58.589870] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.591338] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6701805568, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:58.591392] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=64] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:51:58.592737] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84479822}], scn_array=[cnt:1, {val:1736326318519410000, v:0}], curr_size=586, accum_size=586, time_cost=69422) [2025-01-08 00:51:58.594779] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:58.594948] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=73727, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:58.594972] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2217339}], scn_array=[cnt:1, {val:1736326317803522003, v:0}], curr_size=122, accum_size=244, time_cost=67259) [2025-01-08 00:51:58.594997] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=585, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:58.595058] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:58.595079] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=76899, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:58.595101] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=44, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:58.597971] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=72756, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:58.598012] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18738702}], scn_array=[cnt:1, {val:1736326317803522003, v:0}], curr_size=122, accum_size=723, time_cost=70443) [2025-01-08 00:51:58.598079] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=736, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:58.598313] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:58.598334] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=77698, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:58.598347] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=72, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:58.601854] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48D7-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:58.602600] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4015] clock out of order(cur_ts=1736326318602597, cur_ts_=1736326318566814, delta=35783) [2025-01-08 00:51:58.602927] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:58.609592] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58594][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=26] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:51:58.642682] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=56, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:58.663618] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=37] PNIO [ratelimit] time: 1736326318663614, bytes: 884754657, bw: 0.271892 MB/s, add_ts: 1004311, add_bytes: 286328 [2025-01-08 00:51:58.675515] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=35] gc stale ls task succ [2025-01-08 00:51:58.675690] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:58.675721] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-01-08 00:51:58.675731] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:58.675745] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:58.675759] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193822589366, flush_io_finished_ret_:0, flush_io_finished_round_:1981, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:58.679630] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=32] ts response task statistics(alloc_count=61006, free_count=30502) [2025-01-08 00:51:58.679657] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:51:58.688906] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:58.691879] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDC-0-0] [lt=19] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:58.691913] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDC-0-0] [lt=30] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:58.696658] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=28] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=3, inc_cnt=1, queue_size=3) [2025-01-08 00:51:58.697224] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:51:58.698508] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:58.700096] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=87] current monitor number(seq_=-1) [2025-01-08 00:51:58.706134] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=44] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:4, threads_idle_time:5316495015, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:58.710506] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:58.717741] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A969D-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=845) [2025-01-08 00:51:58.717854] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=92659, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:58.717893] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2251228}], scn_array=[cnt:1, {val:1736326318525513000, v:0}], curr_size=122, accum_size=244, time_cost=115890) [2025-01-08 00:51:58.717941] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=1051, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:58.717959] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1, this=0x2ad8f0fff068, extra_info="log_io_worker_num:4, cb_thread_pool_tg_id:344") [2025-01-08 00:51:58.718958] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=14, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:58.718988] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=117125, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:58.719008] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18496, log_task={header:{begin_lsn:{lsn:2251228}, end_lsn:{lsn:2251350}, log_id:18496, min_scn:{val:1736326318525513000, v:0}, max_scn:{val:1736326318525513000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2251106}, prev_proposal_id:1, committed_end_lsn:{lsn:2251228}, data_checksum:1715074492, accum_checksum:1024903666, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326318601844, freeze_ts:1736326318601844, submit_ts:1736326318601882, flushed_ts:1736326318718888, gen_to_freeze cost time:0, gen_to_submit cost time:38, submit_to_flush cost time:117006}, fs_cb_begin_ts=1736326318718939, log_life_time=117095) [2025-01-08 00:51:58.719060] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=89, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:58.719294] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=150, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:58.719314] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:58.720526] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=25930, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:58.720570] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2238698}], scn_array=[cnt:1, {val:1736326317803522004, v:0}], curr_size=122, accum_size=244, time_cost=112910) [2025-01-08 00:51:58.720597] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=2431, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:58.720608] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1, this=0x2ad8f0fff7a8, extra_info="log_io_worker_num:4, cb_thread_pool_tg_id:344") [2025-01-08 00:51:58.720717] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=19, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:58.720735] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=169032, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:58.720742] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18393, log_task={header:{begin_lsn:{lsn:2238698}, end_lsn:{lsn:2238820}, log_id:18393, min_scn:{val:1736326317803522004, v:0}, max_scn:{val:1736326317803522004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2238576}, prev_proposal_id:1, committed_end_lsn:{lsn:2238698}, data_checksum:940009244, accum_checksum:4171000284, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326318599038, freeze_ts:1736326318599039, submit_ts:1736326318599058, flushed_ts:1736326318720665, gen_to_freeze cost time:1, gen_to_submit cost time:20, submit_to_flush cost time:121607}, fs_cb_begin_ts=1736326318720694, log_life_time=121656) [2025-01-08 00:51:58.720770] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:58.723110] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58661][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D7F-0-0] [lt=35] init point thread id with(&point=0x5572d50e7280, idx_=1317, point=[thread id=58661, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58661) [2025-01-08 00:51:58.723345] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] clock out of order(cur_ts=1736326318723343, cur_ts_=1736326318700194, delta=23149) [2025-01-08 00:51:58.723610] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18738824}], scn_array=[cnt:1, {val:1736326317803522004, v:0}], curr_size=122, accum_size=845, time_cost=95925) [2025-01-08 00:51:58.723641] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=845) [2025-01-08 00:51:58.724032] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=88, avg_log_submit_to_flush_time=85200, avg_log_submit_to_slide_time=89080) [2025-01-08 00:51:58.724746] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=276] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=845, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:58.724804] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=122, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:51:58.724837] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=31] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=53323, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:51:58.724849] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:51:58.724860] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB EXECUTE TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=-2884, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:51:58.724869] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB TOTAL TIME](cur_stat_count=19, stat_interval=5000000, avg (count/cost)=50561, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:51:58.729972] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=68, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:58.730080] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:58.732532] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2251350}], scn_array=[cnt:1, {val:1736326318525513001, v:0}], curr_size=122, accum_size=366, time_cost=14550) [2025-01-08 00:51:58.733983] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58665][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:58.734444] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:51:58.742452] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2238820}], scn_array=[cnt:1, {val:1736326318566814000, v:0}], curr_size=122, accum_size=366, time_cost=21828) [2025-01-08 00:51:58.742571] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=53][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84480408}], scn_array=[cnt:1, {val:1736326318678580000, v:0}], curr_size=122, accum_size=708, time_cost=47982) [2025-01-08 00:51:58.743025] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D160BB030-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326318739496000, start_id=1736326318739496000, end_id=1736326318739496001) [2025-01-08 00:51:58.746737] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=7, avg_log_batch_cnt=1, total_group_log_size=3112, avg_group_log_size=444) [2025-01-08 00:51:58.747013] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2217461}], scn_array=[cnt:1, {val:1736326317803522004, v:0}], curr_size=122, accum_size=366, time_cost=119292) [2025-01-08 00:51:58.747604] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=18219, log_task={header:{begin_lsn:{lsn:2217461}, end_lsn:{lsn:2217583}, log_id:18219, min_scn:{val:1736326317803522004, v:0}, max_scn:{val:1736326317803522004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2217461}, data_checksum:2222604892, accum_checksum:2340559575, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326318627557, freeze_ts:1736326318627559, submit_ts:1736326318627587, flushed_ts:1736326318747573, gen_to_freeze cost time:2, gen_to_submit cost time:30, submit_to_flush cost time:119986}, fs_cb_begin_ts=1736326318747584, log_life_time=120027) [2025-01-08 00:51:58.747857] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=845, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:58.750565] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=39] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5316508409, tenant_id:500}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-01-08 00:51:58.762945] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=33] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=94882, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:58.762981] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43619876}], scn_array=[cnt:1, {val:1736326318503739001, v:0}], curr_size=122, accum_size=5662, time_cost=120047) [2025-01-08 00:51:58.763029] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=941, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:58.766589] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6F5-0-0] [lt=100][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193823680420, cache_obj->added_lc()=false, cache_obj->get_object_id()=13696, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:51:58.769618] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=15, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:58.769641] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=174934, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:58.769651] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=33284, log_task={header:{begin_lsn:{lsn:43619876}, end_lsn:{lsn:43619998}, log_id:33284, min_scn:{val:1736326318503739001, v:0}, max_scn:{val:1736326318503739001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43619876}, data_checksum:667309872, accum_checksum:2843248405, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326318642517, freeze_ts:1736326318642573, submit_ts:1736326318642721, flushed_ts:1736326318769499, gen_to_freeze cost time:56, gen_to_submit cost time:204, submit_to_flush cost time:126778}, fs_cb_begin_ts=1736326318769608, log_life_time=127091) [2025-01-08 00:51:58.769695] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=77, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:58.769884] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=7494, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:58.770933] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58546][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB5C-0-0] [lt=36] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:58.770963] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58546][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB5C-0-0] [lt=23] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:58.771606] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=32] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:58.773937] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=22] ====== tenant freeze timer task ====== [2025-01-08 00:51:58.774920] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:58.775587] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=33] clear dtl interm result cost(us)(clear_cost=14, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:58.776467] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=44][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:58.776497] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=15][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:58.776516] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=66][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:51:58.776379] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=28] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43619998}, readable_log_tail:{lsn:43619998}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23484770, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:58.776570] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=189] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:51:58.776604] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=11] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:51:58.778311] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:58.778410] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:58.778590] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:63, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2047024728, recv_hp_rpc_cnt:5239, recv_np_rpc_cnt:104571, recv_lp_rpc_cnt:0, recv_mysql_cnt:9447, recv_task_cnt:3783, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10904573, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=117136 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1856, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:94667, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736326318339266}{group_id:10, queue_size:0, recv_req_cnt:11284, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2107, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326289381035}{group_id:2, queue_size:0, recv_req_cnt:30348, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11967, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:5008, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:26969, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=281 pcode=0x1502:cnt=230 pcode=0x717:cnt=163 pcode=0x710:cnt=61 pcode=0x150b:cnt=61, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:51:58.779502] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=910] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:15006434, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8257, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3608229, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=17, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:51:58.780308] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=804] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:14745065, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3594759, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:51:58.781429] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [58656][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=11] clear dtl interm result cost(us)(clear_cost=4896, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:51:58.781435] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1126] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:22, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:891999325, recv_hp_rpc_cnt:3792, recv_np_rpc_cnt:38553, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:791184, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=66424 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:68881, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:5, nesting_worker_cnt:0, token_change:1736326318602923}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2226, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326279406832}{group_id:2, queue_size:0, recv_req_cnt:15741, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18870, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=282 pcode=0x51c:cnt=249 pcode=0x717:cnt=167 pcode=0x710:cnt=90 pcode=0x150b:cnt=60, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:51:58.782966] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1529] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:59, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9888181121, recv_hp_rpc_cnt:77, recv_np_rpc_cnt:35526, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2992305, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=25385 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:0, recv_req_cnt:7135, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326269316514}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:152774, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:8, nesting_worker_cnt:0, token_change:1736326318706609}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326302288855}{group_id:19, queue_size:0, recv_req_cnt:414, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326264650105}{group_id:2, queue_size:0, recv_req_cnt:37776, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:11292, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=338 pcode=0x1501:cnt=165 pcode=0x717:cnt=162 pcode=0x1516:cnt=147 pcode=0x150b:cnt=120, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:51:58.788298] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=37] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:51:58.789338] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:58.797024] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC4-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:58.797701] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=74] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:58.798956] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6708097024, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:58.799009] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:51:58.801129] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] update tenant weak read server version(tenant_id=1001, server_version={version:{val:1736326318700194001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1736326318801092}, version_delta=100933) [2025-01-08 00:51:58.801330] INFO [LIB] stat (utility.h:1210) [58640][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B7B-0-0] [lt=39] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:58.801396] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58640][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B7B-0-0] [lt=54] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=854, avg_group_log_size=122) [2025-01-08 00:51:58.801415] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58640][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B7B-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=854) [2025-01-08 00:51:58.801454] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58640][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B7B-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=66, avg_log_submit_to_flush_time=32120, avg_log_submit_to_slide_time=159114) [2025-01-08 00:51:58.801536] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=854, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:58.801567] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18304, miss_cnt:0, cache_read_size:2238942, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:58.805268] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58656][T1_TimerWK4][T1][Y0-0000000000000000-0-0] [lt=43] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:58.806601] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF70-0-0] [lt=69] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326318918662, keepalive_ts:-1}) [2025-01-08 00:51:58.806645] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF70-0-0] [lt=43] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326318918662, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:58.806823] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58604][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCD-0-0] [lt=37] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326318918063, keepalive_ts:-1}) [2025-01-08 00:51:58.806863] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58604][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCD-0-0] [lt=39] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326318918063, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:51:58.823988] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=17] BlockGCTimerTask success(ret=0, cost_time_us=7, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:51:58.829989] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=65, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:58.830022] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=7, avg_log_batch_cnt=1, total_group_log_size=845, avg_group_log_size=120) [2025-01-08 00:51:58.830080] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=845) [2025-01-08 00:51:58.830096] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:58.831728] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC4-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:58.840165] INFO pn_ratelimit (group.c:653) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=31] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:51:58.840408] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58666][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:58.840626] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58667][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:51:58.840689] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=745] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:51:58.851752] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=649809920, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=423931904, wash_ratio=6, tenant_id=1001) [2025-01-08 00:51:58.851848] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=55] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:51:58.851869] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2891, drift_size_=0, pre_drift_size=-6291456, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:58.851932] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1118584832, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2102640640, wash_ratio=6, tenant_id=1002) [2025-01-08 00:51:58.851988] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:51:58.852002] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:51:58.860704] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC431-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:58.889798] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:58.899501] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.899603] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.899761] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.899800] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:58.907140] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AEC3-0-0] [lt=147] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.793514299392700195e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=389, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.610006186249007669e+02) [2025-01-08 00:51:58.919515] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43621119}], scn_array=[cnt:1, {val:1736326318903095000, v:0}], curr_size=589, accum_size=7372, time_cost=11693) [2025-01-08 00:51:58.933011] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] no need to update min/max version(new_version={val:1736326318418972000, v:0}, min_version={val:1736326318418972000, v:0}, max_version={val:1736326318418972000, v:0}) [2025-01-08 00:51:58.933818] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=269] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43621708}, readable_log_tail:{lsn:43621708}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23483060, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:58.973356] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] clock out of order(cur_ts=1736326318973352, cur_ts_=1736326318930462, delta=42890) [2025-01-08 00:51:58.982456] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=49] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:51:58.991381] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:58.996845] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18739190}], scn_array=[cnt:1, {val:1736326318678580002, v:0}], curr_size=122, accum_size=366, time_cost=23029) [2025-01-08 00:51:58.997075] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2217705}], scn_array=[cnt:1, {val:1736326318678580001, v:0}], curr_size=122, accum_size=610, time_cost=23291) [2025-01-08 00:51:59.000363] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6712291328, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:59.005780] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2239064}], scn_array=[cnt:1, {val:1736326318784075001, v:0}], curr_size=122, accum_size=610, time_cost=75247) [2025-01-08 00:51:59.006763] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:51:59.007464] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEE-0-0] [lt=138] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:51:59.007497] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEE-0-0] [lt=24] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=35) [2025-01-08 00:51:59.019418] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=61][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43621708}], scn_array=[cnt:1, {val:1736326318930462000, v:0}], curr_size=589, accum_size=7961, time_cost=85765) [2025-01-08 00:51:59.076562] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BC1-0-0] [lt=20] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:59.091587] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:59.115348] INFO [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1165) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=75] refresh queuing mode(ret=0, tenant_id=1001, stream_cnt=0, update_schema_cnt=0, cost_time=9) [2025-01-08 00:51:59.115516] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1215) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=165] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1001, interval_step=1, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1001, bucket_cnt:0, tenant_mem_user_limit:0, tenant_mem_user_hold:0, cached_node_array:0x2ad8f95d0ee0}) [2025-01-08 00:51:59.116612] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2251838}], scn_array=[cnt:1, {val:1736326318700194003, v:0}], curr_size=122, accum_size=854, time_cost=16305) [2025-01-08 00:51:59.117719] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B7F-0-0] [lt=17][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18396, log_task={header:{begin_lsn:{lsn:2239064}, end_lsn:{lsn:2239186}, log_id:18396, min_scn:{val:1736326318784075001, v:0}, max_scn:{val:1736326318784075001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2238942}, prev_proposal_id:1, committed_end_lsn:{lsn:2239064}, data_checksum:940009244, accum_checksum:1120885306, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326318923408, freeze_ts:1736326318923408, submit_ts:1736326318923421, flushed_ts:1736326319007698, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:84277}, fs_cb_begin_ts=1736326319117703, log_life_time=194295) [2025-01-08 00:51:59.134868] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=45] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:51:59.140464] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F53-0-0] [lt=64] init point thread id with(&point=0x5572d50e73c0, idx_=1322, point=[thread id=58666, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58666) [2025-01-08 00:51:59.150603] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:59.150695] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=90][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:51:59.152246] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=66] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:51:59.152366] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:51:59.152412] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=31] start do ls ha handler(ls_id_array_=[{id:1}, {id:1003}, {id:1001}, {id:1002}]) [2025-01-08 00:51:59.152466] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=84] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:59.152491] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-01-08 00:51:59.152499] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:59.152512] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:59.152525] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193823057876, flush_io_finished_ret_:0, flush_io_finished_round_:3805, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:59.155159] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=99] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:59.155278] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=18] gc stale ls task succ [2025-01-08 00:51:59.157246] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8718D-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:59.157295] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:51:59.157307] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:51:59.158994] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185451-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:59.159157] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:51:59.159177] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-01-08 00:51:59.159244] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:51:59.159263] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:51:59.159276] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193823071425, flush_io_finished_ret_:0, flush_io_finished_round_:1977, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:51:59.160788] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0CB-0-0] [lt=30][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:59.185181] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=94][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18739434}], scn_array=[cnt:1, {val:1736326318678580004, v:0}], curr_size=122, accum_size=610, time_cost=12838) [2025-01-08 00:51:59.191071] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2217949}], scn_array=[cnt:1, {val:1736326318678580003, v:0}], curr_size=122, accum_size=854, time_cost=18886) [2025-01-08 00:51:59.191348] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=85, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:51:59.191391] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:51:59.192488] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=81][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:59.200019] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:59.201011] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=17] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6714388480, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:59.207370] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=90] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:51:59.209429] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=60] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20903, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:51:59.209459] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84481946}], scn_array=[cnt:1, {val:1736326319170191000, v:0}], curr_size=122, accum_size=2246, time_cost=17044) [2025-01-08 00:51:59.209491] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1279, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:51:59.209521] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3519) [2025-01-08 00:51:59.209551] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:51:59.209561] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29531, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:51:59.209576] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=6] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=55, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:51:59.214708] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326309185116, end_time=1736326319214706, dag_status={}) [2025-01-08 00:51:59.214797] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=86] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:59.214862] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=64] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:59.214875] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:59.214883] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:59.214891] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:59.214898] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:59.214957] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:59.214965] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:59.214972] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:51:59.214979] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215051] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=71] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215062] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215071] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215079] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215087] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215095] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215103] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215155] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=51] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215165] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215173] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215287] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=112] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215299] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215308] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215317] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215325] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215333] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215341] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215350] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215358] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215366] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215590] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=170] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215606] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215614] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215622] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215630] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215638] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215698] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=59] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215708] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215716] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215724] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215732] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215740] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215748] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215756] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215764] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215772] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215780] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215788] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215796] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215805] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215813] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215821] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215877] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=55] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215887] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215895] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215903] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215911] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215919] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215927] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215935] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215942] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215951] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215959] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215967] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215974] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215982] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215990] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.215998] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216007] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216014] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216096] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=80] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216109] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216121] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216132] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216145] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216157] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216170] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216183] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216195] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216300] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=104] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216311] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216320] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:51:59.216385] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=62] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:51:59.216394] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:51:59.216402] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:51:59.216408] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:51:59.216413] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:51:59.216468] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=53] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:51:59.216474] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:51:59.216479] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:51:59.216484] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:51:59.216489] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:51:59.216495] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:51:59.238661] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B81-0-0] [lt=73][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18397, log_task={header:{begin_lsn:{lsn:2239186}, end_lsn:{lsn:2239308}, log_id:18397, min_scn:{val:1736326318784075002, v:0}, max_scn:{val:1736326318784075002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2239064}, prev_proposal_id:1, committed_end_lsn:{lsn:2239186}, data_checksum:940009244, accum_checksum:4164939306, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326319117662, freeze_ts:1736326319117662, submit_ts:1736326319117670, flushed_ts:1736326319128381, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:10711}, fs_cb_begin_ts=1736326319238646, log_life_time=120984) [2025-01-08 00:51:59.250282] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEB-0-0] [lt=40][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:51:59.250312] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46496][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEB-0-0] [lt=27][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:59.255799] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=42] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:51:59.255860] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2239308}], scn_array=[cnt:1, {val:1736326318784075003, v:0}], curr_size=122, accum_size=854, time_cost=17094) [2025-01-08 00:51:59.264812] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46389][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CEB-0-0] [lt=71][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:51:59.271937] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:51:59.272256] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:51:59.272502] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=241] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:51:59.277512] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=22] ====== check clog disk timer task ====== [2025-01-08 00:51:59.277574] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=57] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:51:59.278604] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=108] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=6239232) [2025-01-08 00:51:59.278841] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:59.288621] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:59.291504] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:51:59.291576] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44143, [delay/us]:prepare:10, schedule:1871, submit:3441, rt:10640, total:15963) [2025-01-08 00:51:59.291656] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:59.291778] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=103] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44137, [delay/us]: prepare:10, schedule:1871, submit:3441, device:10640, total:15963) [2025-01-08 00:51:59.291793] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:51:59.291870] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:12, bw:60185, [delay/us]:prepare:15, schedule:915, submit:9535, rt:58690, total:69156) [2025-01-08 00:51:59.291884] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:59.292047] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=145] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:5120, avg_iops:12, avg_bw:60177, [delay/us]: prepare:15, schedule:915, submit:9535, device:58690, total:69156) [2025-01-08 00:51:59.292062] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:51:59.292176] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=86] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:5, bw:20057, [delay/us]:prepare:76, schedule:152, submit:1, rt:4010, total:4241) [2025-01-08 00:51:59.292277] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:2, hold_mem:16384 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4216, iops:33, bw:140402, [delay/us]:prepare:7, schedule:2481, submit:7848, rt:15651, total:25989) [2025-01-08 00:51:59.292302] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:1, bw:4011, [delay/us]:prepare:2, schedule:11713, submit:4271, rt:762, total:16748) [2025-01-08 00:51:59.292333] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS TENANT](tenant_id=1002, ref_cnt=6, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49998, free_result_cnt=49998, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:59.292484] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=142] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:5, avg_bw:20055, [delay/us]: prepare:76, schedule:152, submit:1, device:4010, total:4241) [2025-01-08 00:51:59.292508] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4011, [delay/us]: prepare:2, schedule:11713, submit:4271, device:762, total:16748) [2025-01-08 00:51:59.292516] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4216, avg_iops:34, avg_bw:140390, [delay/us]: prepare:7, schedule:2481, submit:7848, device:15651, total:25989) [2025-01-08 00:51:59.292526] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:51:59.292621] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4011, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:949, total:959) [2025-01-08 00:51:59.292815] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=172] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:51:59.292964] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=139] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4009, [delay/us]: prepare:0, schedule:0, submit:600000000, device:949, total:959) [2025-01-08 00:51:59.292989] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:51:59.293005] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:59.293034] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:51:59.293046] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:51:59.293060] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SERVER] net_in=318kB/s, net_out=254kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:51:59.293918] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:59.300407] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=100] ====== check clog disk timer task ====== [2025-01-08 00:51:59.300443] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=31] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:51:59.303993] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=97] PNIO [ratelimit] time: 1736326319303990, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005839, add_bytes: 0 [2025-01-08 00:51:59.306053] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=48] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:59.313615] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=92] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=8172) [2025-01-08 00:51:59.319241] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.319322] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=89] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.348103] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:51:59.348425] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=56] no need to update min/max version(new_version={val:1736326318678580000, v:0}, min_version={val:1736326318678580000, v:0}, max_version={val:1736326318678580000, v:0}) [2025-01-08 00:51:59.356421] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58631][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=64] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:51:59.374511] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5319059343, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:59.374587] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:51:59.374614] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=26] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8bf5f1b40) [2025-01-08 00:51:59.374627] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5319059343, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:51:59.393989] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3983) [2025-01-08 00:51:59.394496] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=99][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:59.395032] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=152] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:256K/60:1837715 disk:256K/60:1837715 t1m2:44K/10:1863029 t1m2g0:44K/10:1863029 t1002m2:156K/38:2305300 t1002m2g0:156K/38:2305300 t1001m2:56K/12:334734 t1001m2g0:56K/12:334734 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:51:59.401533] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6705999872, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:59.407700] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=83] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:51:59.414048] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:59.414121] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=65] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70704720, tenant_id=1001) [2025-01-08 00:51:59.414149] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=19] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:51:59.418433] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=32] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:51:59.418536] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=99] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:59.418555] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=18] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:51:59.418575] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=19] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:59.448718] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35] no need to update min/max version(new_version={val:1736326318678580000, v:0}, min_version={val:1736326318678580000, v:0}, max_version={val:1736326318678580000, v:0}) [2025-01-08 00:51:59.542042] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.548854] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326318678580003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=870265, in_cluster_service=true, cluster_version={val:1736326318678580000, v:0}, min_cluster_version={val:1736326318678580000, v:0}, max_cluster_version={val:1736326318678580000, v:0}, get_cluster_version_err=0, cluster_version_delta=870265, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18714, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=547466372, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:51:59.548991] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=121] no need to update min/max version(new_version={val:1736326318678580000, v:0}, min_version={val:1736326318678580000, v:0}, max_version={val:1736326318678580000, v:0}) [2025-01-08 00:51:59.602208] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6705999872, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:59.612100] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=38] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1541344, replace_num=15728) [2025-01-08 00:51:59.612131] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:51:59.621684] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BC3-0-0] [lt=122] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:51:59.671333] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1736326319671329, bytes: 884919586, bw: 0.156084 MB/s, add_ts: 1007715, add_bytes: 164929 [2025-01-08 00:51:59.697558] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D53-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:51:59.699915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:59.683, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.710788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:59.702, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.711879] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:51:59.757481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=127][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:59.747, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.757605] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58665][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=72] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:51:59.826059] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:51:59.826679] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4016] task executed too delay(delay_time=294265, *p_task_={this:0x2ad8950efb10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8950efdf0, time_wheel:0x2ad893bed990, thread_pool:0x2ad893bed170, is_running:0, time_interval:250.00ms, expected_run_time:00:51:59.775, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.826797] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=283][errcode=-4016] task executed too delay(delay_time=378140, *p_task_={this:0x2ad8efffe030, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8efffe310, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:59.933, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.826907] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=64][errcode=-4016] task executed too delay(delay_time=357849, *p_task_={this:0x2ad8f0dfe7f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfead0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:59.952, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.826938] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] task executed too delay(delay_time=327209, *p_task_={this:0x2ad8f0dff570, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:51:59.989, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.826962] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] task executed too delay(delay_time=317063, *p_task_={this:0x2ad8f0dff950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dffc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:59.997, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.826986] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] task executed too delay(delay_time=306872, *p_task_={this:0x2ad8f0dfc950, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfcc30, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:52:00.510, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.827011] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] task executed too delay(delay_time=306885, *p_task_={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:250.00ms, expected_run_time:00:51:59.752, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.827037] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] task executed too delay(delay_time=225194, *p_task_={this:0x2ad8f0dfe410, caller:election_proposer.cpp:register_renew_lease_task_:270, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8f0dfe6f0, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:0, time_interval:500.00ms, expected_run_time:00:52:00.919, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.827426] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=97] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.827477] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.827501] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.827535] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.830736] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=412, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:51:59.831695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50378][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] task scheduled too delay(scheduled_delay=324025, expected_run_ts_us_=1736326319507666, ObClockGenerator::getRealClock()=1736326319831692, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05d30, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8efe559b0, time_wheel:0x2ad8cf1f7990, thread_pool:0x2ad8cf1f7170, is_running:1, time_interval:1.00s, expected_run_time:00:51:59.507, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.832026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46270][T1_CoordTR][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] task scheduled too delay(scheduled_delay=225557, expected_run_ts_us_=1736326319606465, ObClockGenerator::getRealClock()=1736326319832024, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3a70, caller:ob_failure_detector.cpp:mtl_start:94, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x2ad8950f3d50, time_wheel:0x2ad8879cfd10, thread_pool:0x2ad893be57c0, is_running:1, time_interval:1.00s, expected_run_time:00:51:59.606, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.832121] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] no need to update min/max version(new_version={val:1736326318678580000, v:0}, min_version={val:1736326318678580000, v:0}, max_version={val:1736326318678580000, v:0}) [2025-01-08 00:51:59.833001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50406][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] task scheduled too delay(scheduled_delay=528131, expected_run_ts_us_=1736326319304808, ObClockGenerator::getRealClock()=1736326319832999, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe46190, caller:ob_tenant_freezer.cpp:start:133, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_494, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe46470, time_wheel:0x2ad8cf0e7990, thread_pool:0x2ad8dd4ab4f0, is_running:1, time_interval:2.00s, expected_run_time:00:51:59.304, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.833423] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:51:59.834064] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] success to update min reserved snapshot(reserved_snapshot=1736324515701616000, duration=1800, snapshot_gc_ts=1736326315701616000) [2025-01-08 00:51:59.834118] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58658][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=0] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=2, size_used=0, mem_used=16637952) [2025-01-08 00:51:59.834162] INFO [LIB] stat (utility.h:1210) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:51:59.834233] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:51:59.834869] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=163] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6699708416, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:51:59.835151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [46272][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] task scheduled too delay(scheduled_delay=428683, expected_run_ts_us_=1736326319406463, ObClockGenerator::getRealClock()=1736326319835148, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8950f3690, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8950f3970, time_wheel:0x2ad893be9990, thread_pool:0x2ad893be9170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:59.406, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.836616] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50530][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] task scheduled too delay(scheduled_delay=428082, expected_run_ts_us_=1736326319408530, ObClockGenerator::getRealClock()=1736326319836614, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8a0dff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x2ad8a0dff850, time_wheel:0x2ad8cf1f9990, thread_pool:0x2ad8cf1f9170, is_running:1, time_interval:250.00ms, expected_run_time:00:51:59.408, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.836912] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:59.837134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50528][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=189][errcode=0] task scheduled too delay(scheduled_delay=487776, expected_run_ts_us_=1736326319349355, ObClockGenerator::getRealClock()=1736326319837132, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8f0dfd950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8f0dfdc30, time_wheel:0x2ad8cf0a1990, thread_pool:0x2ad8cf0a1170, is_running:1, time_interval:100.00ms, expected_run_time:00:51:59.349, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.838434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50380][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] task scheduled too delay(scheduled_delay=430767, expected_run_ts_us_=1736326319407663, ObClockGenerator::getRealClock()=1736326319838431, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8efe05950, caller:ob_failure_detector.cpp:mtl_start:89, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x2ad8efe05c30, time_wheel:0x2ad8dacf7c20, thread_pool:0x2ad8dacf7400, is_running:1, time_interval:100.00ms, expected_run_time:00:51:59.407, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.839672] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=70] ====== checkpoint timer task ====== [2025-01-08 00:51:59.840484] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=32] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1415565, clean_num=31457) [2025-01-08 00:51:59.841117] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:51:59.841494] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=21] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:1, active_tx_count:1, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:59.841554] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=59] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1736326319313412000, v:0}) [2025-01-08 00:51:59.841716] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=27] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329284252000, latest_log_ts:{val:1736326312645069001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:59.841895] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=172] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:59.842286] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=345] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:59.842314] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] get rec log scn(service_type_=0, rec_log_ts={val:1736326312645069001, v:0}) [2025-01-08 00:51:59.842325] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:51:59.842333] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:51:59.842348] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:59.842368] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:59.842382] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:59.842406] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:59.842429] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:59.842440] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:59.843706] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:15273457998, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:59.843932] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:51:59.844029] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=212] recycle one thread(this=0x5572d527f900, total=6, remain=5) [2025-01-08 00:51:59.844101] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=69] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:15273457998, tenant_id:500}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:59.844214] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=110] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15202661770, tenant_id:1}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:59.844868] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=514] recycle one thread(this=0x2ad8881fc3b0, total=7, remain=6) [2025-01-08 00:51:59.844892] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=22] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15202661770, tenant_id:1}, cur_thread_count=7, cnt=-1, new_thread_count=6) [2025-01-08 00:51:59.844912] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:3, threads_idle_time:8002792287, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:59.845095] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] recycle one thread(this=0x2ad8c0eb0e70, total=6, remain=5) [2025-01-08 00:51:59.845124] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=28] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:3, threads_idle_time:8002792287, tenant_id:1002}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:51:59.846489] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:51:59.850380] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326319418178000, start_id=1736326319418178001, end_id=1736326319418178002) [2025-01-08 00:51:59.852667] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] clock out of order(cur_ts=1736326319852664, cur_ts_=1736326319418178, delta=434486) [2025-01-08 00:51:59.852697] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:51:59.855726] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDD-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:51:59.855816] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDD-0-0] [lt=84] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:51:59.855336] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=8] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:51:59.857427] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=2084] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:51:59.862109] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=106] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:51:59.868129] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=77] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:51:59.869339] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC4-0-0] [lt=24] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:59.869439] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC4-0-0] [lt=81] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:59.870835] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=120] report RowHolderMapper summary info(count=1, bkt_cnt=248) [2025-01-08 00:51:59.870884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [50382][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] task scheduled too delay(scheduled_delay=117954, expected_run_ts_us_=1736326319752927, ObClockGenerator::getRealClock()=1736326319870882, ret=0, ret="OB_SUCCESS", *this={this:0x2ad8effff570, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:12, func_shared_ptr_.ptr:0x2ad8effff850, time_wheel:0x2ad8856872f0, thread_pool:0x2ad8856f37b0, is_running:1, time_interval:250.00ms, expected_run_time:00:51:59.752, task_priority:1, with_handle_protected:1}) [2025-01-08 00:51:59.871068] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [58658][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:51:59.872521] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=72] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:59.872849] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE95D-0-0] [lt=109] get number(ret=0, service_type_=0, range=1, base_id=1736326319864970000, start_id=1736326319864970000, end_id=1736326319864970001) [2025-01-08 00:51:59.876450] WDIAG [PALF] runTimerTask (log_updater.cpp:109) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] update_palf cost too much time(ret=0, cost_time_us=604949, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}, log_alloc_mgr_:{flying_log_task:2, flying_meta_task:0}}) [2025-01-08 00:51:59.877787] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=65] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=96715, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:51:59.877828] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43623597}], scn_array=[cnt:1, {val:1736326319313412000, v:0}], curr_size=589, accum_size=9850, time_cost=563975) [2025-01-08 00:51:59.877851] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9850) [2025-01-08 00:51:59.877904] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=1190, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:51:59.877951] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9261) [2025-01-08 00:51:59.878020] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=74, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:51:59.878043] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=6360, avg_group_log_size=530) [2025-01-08 00:51:59.878097] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=48] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=49, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:51:59.878490] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4122) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A71-0-0] [lt=61][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="192.168.159.137:2882", server="192.168.159.139:2882", update_func={old_end_lsn:{lsn:18739190}, new_end_lsn:{lsn:18739312}, old_advance_time_us:1736326318836873, new_ack_time_us:1736326319878486, advance delay(us):1041613}) [2025-01-08 00:51:59.878578] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A71-0-0] [lt=83] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=732) [2025-01-08 00:51:59.881094] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84482654}], scn_array=[cnt:1, {val:1736326319240539001, v:0}], curr_size=122, accum_size=2954, time_cost=486686) [2025-01-08 00:51:59.881118] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2954) [2025-01-08 00:51:59.881151] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=178] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18743652, this={dio_aligned_buf:{buf_write_offset:478, buf_padding_size:3618, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:33040, truncate_used_ts:6791}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=18743296, buf_len=122, total_write_size=12978, total_write_size_after_dio=344064, ob_pwrite_used_ts=2721037, count=80, each_pwrite_cost=34012) [2025-01-08 00:51:59.881269] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=113] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=107732, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:51:59.881316] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18739556}], scn_array=[cnt:1, {val:1736326319170191000, v:0}], curr_size=122, accum_size=732, time_cost=608552) [2025-01-08 00:51:59.881335] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=732) [2025-01-08 00:51:59.881352] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=807, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:51:59.881458] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=73, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:51:59.881481] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=53, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:51:59.882447] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:51:59.883088] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:59.883303] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=30] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326314412287000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:59.886144] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2222167, this={dio_aligned_buf:{buf_write_offset:2257, buf_padding_size:1839, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:23081, truncate_used_ts:5225}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2220032, buf_len=122, total_write_size=9732, total_write_size_after_dio=335872, ob_pwrite_used_ts=2701124, count=80, each_pwrite_cost=33764) [2025-01-08 00:51:59.886156] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:51:59.886244] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=130975, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:51:59.886281] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2218071}], scn_array=[cnt:1, {val:1736326319170191000, v:0}], curr_size=122, accum_size=976, time_cost=613376) [2025-01-08 00:51:59.886313] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=976) [2025-01-08 00:51:59.886332] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=533, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:51:59.886729] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=976) [2025-01-08 00:51:59.886803] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=70] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=39, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:51:59.886895] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48E5-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:51:59.886917] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48E5-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:51:59.886931] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48E5-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:51:59.887498] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB69-0-0] [lt=34] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:59.887530] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB69-0-0] [lt=25] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:59.887765] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A1D-0-0] [lt=42] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=2389, avg_log_gen_to_submit_time=2493, avg_log_submit_to_flush_time=20850, avg_log_submit_to_slide_time=86302) [2025-01-08 00:51:59.887806] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A1D-0-0] [lt=39][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=45115, log_task={header:{begin_lsn:{lsn:84482068}, end_lsn:{lsn:84482654}, log_id:45115, min_scn:{val:1736326319240539000, v:0}, max_scn:{val:1736326319240539000, v:0}, data_len:530, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84482068}, data_checksum:2731464762, accum_checksum:536625222, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326319243695, freeze_ts:1736326319243702, submit_ts:1736326319243817, flushed_ts:1736326319255773, gen_to_freeze cost time:7, gen_to_submit cost time:122, submit_to_flush cost time:11956}, fs_cb_begin_ts=1736326319887739, log_life_time=644044) [2025-01-08 00:51:59.887983] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=87] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:59.887999] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=15] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:51:59.888134] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84994-0-0] [lt=65] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:59.888269] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84994-0-0] [lt=131] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:51:59.888990] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAED1-0-0] [lt=26] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:51:59.889012] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAED1-0-0] [lt=18] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:51:59.891402] INFO [LIB] stat (utility.h:1210) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A96AB-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=9, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:51:59.891422] INFO [LIB] stat (utility.h:1210) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A96AB-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=135319, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:51:59.891432] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A96AB-0-0] [lt=8][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=18224, log_task={header:{begin_lsn:{lsn:2218071}, end_lsn:{lsn:2218193}, log_id:18224, min_scn:{val:1736326319170191000, v:0}, max_scn:{val:1736326319170191000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2218071}, data_checksum:2222604892, accum_checksum:133718679, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326319272534, freeze_ts:1736326319272538, submit_ts:1736326319272681, flushed_ts:1736326319886721, gen_to_freeze cost time:4, gen_to_submit cost time:147, submit_to_flush cost time:614040}, fs_cb_begin_ts=1736326319891390, log_life_time=618856) [2025-01-08 00:51:59.892502] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Slow write(duration=54386, write_len=4096) [2025-01-08 00:51:59.892526] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=22] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8833, offset=2509631}, end_cursor:ObLogCursor{file_id=1, log_id=8834, offset=2509936}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894cfc050, len:1217, log_data_len:305, seq:8833, flush_finish:false, flush_ret:0}) [2025-01-08 00:51:59.892765] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E78-0-0] [lt=68] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:51:59.892798] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E78-0-0] [lt=17] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326318678580004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326318678580004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:51:59.893403] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=36][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326319240539000, v:0}, log_ts:{val:1736326319240539000, v:0}, lsn:{lsn:84482124}, submit_ts:1736326319243628}, this:0x2ad89c0a3bd8, is_inited_:true, trans_id:{txid:28256}, ls_id:{id:1}, ctx:0x2ad89c0a01d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=649771, ctx_lock_wait_time=1) [2025-01-08 00:51:59.893665] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=70] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3983, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:59.894321] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=132] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=610, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:59.894618] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326318700194002, v:0}, new_max_version={val:1736326318700194002, v:0}, last_min_version={val:1736326318700194001, v:0}, last_max_version={val:1736326318700194001, v:0}, total_time_us=653374, delta_us=1194421) [2025-01-08 00:51:59.894651] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326318700194002, v:0}, min_version={val:1736326318700194002, v:0}, max_version={val:1736326318700194002, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:51:59.894752] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1736326319036658000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84482654}, palf_committed_end_scn_:{val:1736326319240539000, v:0}, last_check_scn_:{val:1736326319240539001, v:0}, max_applied_cb_scn_:{val:1736326319036658000, v:0}}) [2025-01-08 00:51:59.894779] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84482776}, readable_log_tail:{lsn:84482776}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49726760, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:51:59.894982] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=124] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:51:59.896912] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.896952] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.897039] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:51:59.897146] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=6575, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:51:59.897164] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:51:59.897171] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:51:59.897237] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=64] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=3189, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:51:59.897267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.897294] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:51:59.897375] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=39, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:51:59.901851] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58667][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=43] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:51:59.902730] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DAFB-0-0] [lt=7] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=488) [2025-01-08 00:51:59.902773] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DAFB-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:51:59.902786] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DAFB-0-0] [lt=12] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=220553, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:51:59.902795] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DAFB-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=26, avg_log_submit_to_flush_time=28498, avg_log_submit_to_slide_time=250647) [2025-01-08 00:51:59.902808] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DAFB-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18398, log_task={header:{begin_lsn:{lsn:2239308}, end_lsn:{lsn:2239430}, log_id:18398, min_scn:{val:1736326318784075003, v:0}, max_scn:{val:1736326318784075003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2239186}, prev_proposal_id:1, committed_end_lsn:{lsn:2239308}, data_checksum:940009244, accum_checksum:2505984575, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326319238614, freeze_ts:1736326319238614, submit_ts:1736326319238629, flushed_ts:1736326319255910, gen_to_freeze cost time:0, gen_to_submit cost time:15, submit_to_flush cost time:17281}, fs_cb_begin_ts=1736326319902762, log_life_time=664148) [2025-01-08 00:51:59.903067] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=488, hit_cnt=4, read_cnt=4, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:59.903165] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18308, miss_cnt:0, cache_read_size:2239430, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:59.903579] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=184] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=713, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:51:59.903602] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:51:59.907904] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [58654][T1002_TimerWK4][T1002][Y0-0000000000000000-0-0] [lt=78] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:51:59.916216] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC5-0-0] [lt=54][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:59.916395] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A75-0-0] [lt=108] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=25, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:51:59.916469] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A75-0-0] [lt=72] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=116402, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:51:59.916486] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A75-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=140, avg_log_submit_to_flush_time=112159, avg_log_submit_to_slide_time=119584) [2025-01-08 00:51:59.916500] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A75-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=19865, log_task={header:{begin_lsn:{lsn:18739556}, end_lsn:{lsn:18739678}, log_id:19865, min_scn:{val:1736326319170191000, v:0}, max_scn:{val:1736326319170191000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18739556}, data_checksum:667309872, accum_checksum:2137473407, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326319272147, freeze_ts:1736326319272155, submit_ts:1736326319272245, flushed_ts:1736326319881414, gen_to_freeze cost time:8, gen_to_submit cost time:98, submit_to_flush cost time:609169}, fs_cb_begin_ts=1736326319916382, log_life_time=644235) [2025-01-08 00:51:59.916673] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=732, hit_cnt=6, read_cnt=6, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:59.916972] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=227] ====== checkpoint timer task ====== [2025-01-08 00:51:59.917807] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=20] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:59.917847] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:51:59.917945] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329254996000, latest_log_ts:{val:1736326312592647001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:51:59.918070] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=122] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329254996000, latest_log_ts:{val:1736326312592647001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:51:59.918242] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=163] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:51:59.918257] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1736326312592647001, v:0}) [2025-01-08 00:51:59.918311] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=53] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:51:59.918319] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:51:59.918332] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:59.918405] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:51:59.918425] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:51:59.918443] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:51:59.918515] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=71] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:51:59.918525] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:51:59.919664] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E78-0-0] [lt=9] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=3, total_release_part_ctx_count=67) [2025-01-08 00:51:59.919699] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E78-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.793514299392700195e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=389, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.610006186249007669e+02) [2025-01-08 00:51:59.924264] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC5-0-0] [lt=66][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:59.925775] INFO [LIB] stat (utility.h:1210) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F56-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=42, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:51:59.925814] INFO [LIB] stat (utility.h:1210) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F56-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=118296, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:51:59.925828] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F56-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=53862, avg_log_gen_to_submit_time=53955, avg_log_submit_to_flush_time=70566, avg_log_submit_to_slide_time=76600) [2025-01-08 00:51:59.925895] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F56-0-0] [lt=64][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=33291, log_task={header:{begin_lsn:{lsn:43623597}, end_lsn:{lsn:43624186}, log_id:33291, min_scn:{val:1736326319313412000, v:0}, max_scn:{val:1736326319313412000, v:0}, data_len:533, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43623597}, data_checksum:3898126263, accum_checksum:2209527610, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326319313499, freeze_ts:1736326319313503, submit_ts:1736326319313602, flushed_ts:1736326319877935, gen_to_freeze cost time:4, gen_to_submit cost time:103, submit_to_flush cost time:564333}, fs_cb_begin_ts=1736326319925712, log_life_time=612213) [2025-01-08 00:51:59.928153] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326319313412000, v:0}, log_ts:{val:1736326319313412000, v:0}, lsn:{lsn:43623653}, submit_ts:1736326319313493}, this:0x2ad8fb7086d8, is_inited_:true, trans_id:{txid:67354}, ls_id:{id:1}, ctx:0x2ad8fb704cd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=614655, ctx_lock_wait_time=1) [2025-01-08 00:51:59.928458] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4188, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:59.929062] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] get_max_applied_scn(scn={val:1736326319140657000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43624186}, palf_committed_end_scn_:{val:1736326319313412000, v:0}, last_check_scn_:{val:1736326319418178000, v:0}, max_applied_cb_scn_:{val:1736326319140657000, v:0}}) [2025-01-08 00:51:59.931966] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:51:59.933673] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326319931368000, start_id=1736326319931368000, end_id=1736326319931368001) [2025-01-08 00:51:59.934039] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=77] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=7, avg_log_batch_cnt=1, total_group_log_size=2205, avg_group_log_size=315) [2025-01-08 00:51:59.934421] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:51:59.934443] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] try gc retain ctx [2025-01-08 00:51:59.934451] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:59.934470] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326319140657000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326314453572000, v:0}, keep_alive_scn_:{val:1736326314657020001, v:0}, update_ts_:1736326319931368}) [2025-01-08 00:51:59.935361] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329284252000, latest_log_ts:{val:1736326312645069001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:59.936303] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] LogLoopThread round_cost_time(us)(round_cost_time=38) [2025-01-08 00:51:59.938254] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17] LogLoopThread round_cost_time(us)(round_cost_time=12) [2025-01-08 00:51:59.938284] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:45117, max_lsn:84483321, max_scn:{val:1736326319931368000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84482776}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42540281}}, last_submit_lsn:{lsn:84482776}, last_submit_end_lsn:{lsn:84483321}, last_submit_log_id:45117, last_submit_log_pid:1, max_flushed_lsn:{lsn:84482654}, max_flushed_end_lsn:{lsn:84482776}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84482654}, last_slide_log_id:45115, last_slide_scn:{val:1736326319240539000, v:0}, last_slide_lsn:{lsn:84482068}, last_slide_end_lsn:{lsn:84482654}, last_slide_log_pid:1, last_slide_log_accum_checksum:536625222, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:45116, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84482776}, readable_log_tail:{lsn:84482776}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49726760, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:51:59.938715] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=425] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326319881392, last_flushed_end_lsn:{lsn:84482776}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326319887731, last_flushed_end_lsn:{lsn:84482654}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326319916082, last_flushed_end_lsn:{lsn:84482654}}]) [2025-01-08 00:51:59.938741] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=45116, log_task={header:{begin_lsn:{lsn:84482654}, end_lsn:{lsn:84482776}, log_id:45116, min_scn:{val:1736326319240539001, v:0}, max_scn:{val:1736326319240539001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84482068}, data_checksum:667309872, accum_checksum:238212201, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1736326319393806, freeze_ts:1736326319393873, submit_ts:1736326319393977, flushed_ts:1736326319881390, gen_to_freeze cost time:67, gen_to_submit cost time:171, submit_to_flush cost time:487413}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326319881392, last_flushed_end_lsn:{lsn:84482776}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326319887731, last_flushed_end_lsn:{lsn:84482654}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326319916082, last_flushed_end_lsn:{lsn:84482654}}]) [2025-01-08 00:51:59.938899] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:51:59.941090] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=44] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:51:59.942985] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:51:59.943074] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326315839366000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:51:59.948747] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43624308}], scn_array=[cnt:1, {val:1736326319922063000, v:0}], curr_size=1135, accum_size=1257, time_cost=23499) [2025-01-08 00:51:59.949163] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84482776}], scn_array=[cnt:1, {val:1736326319931368000, v:0}], curr_size=545, accum_size=545, time_cost=14787) [2025-01-08 00:51:59.951695] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=54, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:51:59.951797] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=5, avg_log_batch_cnt=1, total_group_log_size=610, avg_group_log_size=122) [2025-01-08 00:51:59.951819] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=610) [2025-01-08 00:51:59.951833] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:51:59.954268] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=41] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:51:59.954294] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] try gc retain ctx [2025-01-08 00:51:59.954302] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:51:59.954345] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326319036658000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326309543479001, v:0}, keep_alive_scn_:{val:1736326314648302000, v:0}, update_ts_:1736326319954115}) [2025-01-08 00:51:59.955521] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=59] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329254996000, latest_log_ts:{val:1736326312592647001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:51:59.961706] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=23] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12796379136, fast_delete_log_mode=false) [2025-01-08 00:51:59.961755] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326319418178001, v:0}, log_ts:{val:1736326319418178001, v:0}, lsn:{lsn:18739790}, submit_ts:1736326319850470}, this:0x2ad8c34736d8, is_inited_:true, trans_id:{txid:7578}, ls_id:{id:1}, ctx:0x2ad8c346fcd0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=111280, ctx_lock_wait_time=1) [2025-01-08 00:51:59.964401] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48E7-0-0] [lt=84] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=20, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:51:59.964458] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48E7-0-0] [lt=55] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=43042, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:51:59.964472] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48E7-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=16, avg_log_submit_to_flush_time=24489, avg_log_submit_to_slide_time=52282) [2025-01-08 00:51:59.965492] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=976, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:51:59.965529] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18410, miss_cnt:0, cache_read_size:2252204, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:51:59.966121] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2218193}], scn_array=[cnt:1, {val:1736326319418178001, v:0}], curr_size=122, accum_size=122, time_cost=12365) [2025-01-08 00:51:59.969521] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=26] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:51:59.971907] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326319970373000, start_id=1736326319970373000, end_id=1736326319970373001) [2025-01-08 00:51:59.975328] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC434-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:51:59.980304] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=103] [APPLY STAT CB APPEND COST TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=109, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:51:59.980333] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=29] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=87761, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:51:59.980342] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:51:59.980350] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=-11375, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:51:59.980357] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=33, stat_interval=5000000, avg (count/cost)=76495, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:51:59.984655] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B83-0-0] [lt=149] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:51:59.984708] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B83-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=4, avg_log_batch_cnt=0, total_group_log_size=479, avg_group_log_size=119) [2025-01-08 00:51:59.988656] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [58658][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=39] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:51:59.994373] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:342) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17CBB96D-0-0] [lt=8][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:28257}, ref_cnt:1, state:"RUNNING", commit_version:{val:1736326319931368000, v:0}, start_scn:{val:1736326319931368000, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout_ts:1736326349891565, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326319931368000, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, data_trans_id_:{txid:28257}, data_sql_sequence_:{branch:0, seq:3}, read_latest_:false, read_uncommitted_:false, scn_:{val:1736326319931368000, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1736326319931368000, v:0}}) [2025-01-08 00:51:59.995710] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43625443}], scn_array=[cnt:1, {val:1736326319931368000, v:0}], curr_size=589, accum_size=1846, time_cost=37884) [2025-01-08 00:51:59.996576] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=22861, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:51:59.996614] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2239430}], scn_array=[cnt:1, {val:1736326319864970000, v:0}], curr_size=113, accum_size=967, time_cost=11805) [2025-01-08 00:51:59.996639] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=967) [2025-01-08 00:51:59.996659] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=4610, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:51:59.996789] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=45, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:51:59.997121] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=119][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2218315}], scn_array=[cnt:1, {val:1736326319418178002, v:0}], curr_size=112, accum_size=234, time_cost=27133) [2025-01-08 00:51:59.998294] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2252204}], scn_array=[cnt:1, {val:1736326319877567000, v:0}], curr_size=122, accum_size=122, time_cost=32870) [2025-01-08 00:52:00.023467] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58669][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:00.023541] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58669][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F58-0-0] [lt=38] init point thread id with(&point=0x5572d50e7480, idx_=1325, point=[thread id=58669, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58669) [2025-01-08 00:52:00.023712] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326319922063000, v:0}, log_ts:{val:1736326319922063000, v:0}, lsn:{lsn:43624364}, submit_ts:1736326319922998}, this:0x2ad8fb6a32d8, is_inited_:true, trans_id:{txid:67355}, ls_id:{id:1}, ctx:0x2ad8fb69f8d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=100709, ctx_lock_wait_time=2) [2025-01-08 00:52:00.024077] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:52:00.041340] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.049456] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6712291328, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:00.049557] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:52:00.049730] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326319140657000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=909021, in_cluster_service=true, cluster_version={val:1736326318903095000, v:0}, min_cluster_version={val:1736326318903095000, v:0}, max_cluster_version={val:1736326318903095000, v:0}, get_cluster_version_err=0, cluster_version_delta=1146583, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=19143, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736326314422615000, v:0}, local_cluster_delta=5627063, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:52:00.054986] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=253] get number(ret=0, service_type_=0, range=1, base_id=1736326320048124000, start_id=1736326320048124000, end_id=1736326320048124001) [2025-01-08 00:52:00.057921] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58669][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F59-0-0] [lt=23][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=33294, log_task={header:{begin_lsn:{lsn:43625443}, end_lsn:{lsn:43626032}, log_id:33294, min_scn:{val:1736326319931368000, v:0}, max_scn:{val:1736326319931368000, v:0}, data_len:533, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:43624186}, data_checksum:3053252359, accum_checksum:2452931519, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326319945324, freeze_ts:1736326319956897, submit_ts:1736326319956988, flushed_ts:1736326319998131, gen_to_freeze cost time:11573, gen_to_submit cost time:11664, submit_to_flush cost time:41143}, fs_cb_begin_ts=1736326320057866, log_life_time=112542) [2025-01-08 00:52:00.058265] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326319931368000, v:0}, log_ts:{val:1736326319931368000, v:0}, lsn:{lsn:43625499}, submit_ts:1736326319945316}, this:0x2ad8fb6c0b58, is_inited_:true, trans_id:{txid:67356}, ls_id:{id:1}, ctx:0x2ad8fb6bd150, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=112945, ctx_lock_wait_time=1) [2025-01-08 00:52:00.061745] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58660][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D87-0-0] [lt=27] init point thread id with(&point=0x5572d50e7240, idx_=1316, point=[thread id=58660, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58660) [2025-01-08 00:52:00.061926] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58660][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D87-0-0] [lt=157][errcode=-4389] log_task life cost too much time(palf_id=1, self="192.168.159.137:2882", log_id=45117, log_task={header:{begin_lsn:{lsn:84482776}, end_lsn:{lsn:84483321}, log_id:45117, min_scn:{val:1736326319931368000, v:0}, max_scn:{val:1736326319931368000, v:0}, data_len:489, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:84482654}, data_checksum:1717768781, accum_checksum:512009887, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326319933815, freeze_ts:1736326319933822, submit_ts:1736326319933953, flushed_ts:1736326319949437, gen_to_freeze cost time:7, gen_to_submit cost time:138, submit_to_flush cost time:15484}, fs_cb_begin_ts=1736326320061917, log_life_time=128102) [2025-01-08 00:52:00.063001] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=62][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326319931368000, v:0}, log_ts:{val:1736326319931368000, v:0}, lsn:{lsn:84482832}, submit_ts:1736326319933806}, this:0x2ad89c064758, is_inited_:true, trans_id:{txid:28257}, ls_id:{id:1}, ctx:0x2ad89c060d50, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=129191, ctx_lock_wait_time=1) [2025-01-08 00:52:00.064185] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A79-0-0] [lt=17][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.159.137:2882", log_id=18225, log_task={header:{begin_lsn:{lsn:2218193}, end_lsn:{lsn:2218315}, log_id:18225, min_scn:{val:1736326319418178001, v:0}, max_scn:{val:1736326319418178001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:2218193}, data_checksum:2222604892, accum_checksum:1798320101, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1736326319951617, freeze_ts:1736326319951625, submit_ts:1736326319951775, flushed_ts:1736326319968993, gen_to_freeze cost time:8, gen_to_submit cost time:158, submit_to_flush cost time:17218}, fs_cb_begin_ts=1736326320064176, log_life_time=112559) [2025-01-08 00:52:00.067999] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BED8-0-0] [lt=46] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:52:00.075500] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=16] Slow write(duration=24706, write_len=4096) [2025-01-08 00:52:00.075548] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=46] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8834, offset=2509936}, end_cursor:ObLogCursor{file_id=1, log_id=8835, offset=2510233}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d00050, len:912, log_data_len:297, seq:8834, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:00.075711] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18740279}], scn_array=[cnt:1, {val:1736326319418178002, v:0}], curr_size=122, accum_size=723, time_cost=16861) [2025-01-08 00:52:00.076455] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E78-0-0] [lt=18] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:00.076572] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E78-0-0] [lt=51] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326319170191000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326319170191000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:00.076757] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=117][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43626578}], scn_array=[cnt:1, {val:1736326320048124000, v:0}], curr_size=589, accum_size=2981, time_cost=16841) [2025-01-08 00:52:00.102396] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4015] clock out of order(cur_ts=1736326320102392, cur_ts_=1736326320077497, delta=24895) [2025-01-08 00:52:00.106522] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CB-0-0] [lt=74] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:00.106565] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CB-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326319201361000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326319201361000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:00.109824] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46493][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBA1-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193825023655, cache_obj->added_lc()=false, cache_obj->get_object_id()=13697, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:00.112951] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] update reserved snapshot finished(cost_ts=278889, reserved_snapshot=1736324515701616000) [2025-01-08 00:52:00.116246] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=40][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922106e48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:136704} write_epoch=1, prev=this=0x2ad9220f9dd0 trans_version={val:1736325659490935000, v:0} scn={val:1736325659490935000, v:0} tx_id={txid:22448} prev=0x2ad9220f9a50 next=(nil) modify_count=4 acc_checksum=23701767 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:34467} write_epoch=3, snapshot_version={val:1736326320073024000, v:0}, *this={this=0x2ad9220b7d10 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220f9dd0 latest_compact_node=(nil) max_trans_version={val:1736325659490935000, v:0} max_trans_id=22448 max_elr_trans_version={val:1736325659490935000, v:0} max_elr_trans_id=22448 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325659490935000, v:0} min_modify_scn={val:1736324459170693000, v:0}}) [2025-01-08 00:52:00.116308] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=61][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922106e48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:136704} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326349956382, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326349956382, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326320073024000, v:0}, tx_id:{txid:23718}, scn:{branch:0, seq:136701}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:23718}, tx_desc:{this:0x2ad8b86aa860, tx_id:{txid:23718}, state:1, addr:"192.168.159.138:2882", tenant_id:1001, session_id:3221815648, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326320106877, commit_ts:-1, finish_ts:-1, timeout_us:29999954, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326320088829, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6bd150, mem_ctx:0x2ad8fb6be6d0, tx_scn:{branch:0, seq:136704}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:00.116428] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=119][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:23718}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:136704}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:00.116447] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"BIGINT UNSIGNED":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:23718} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:00.116480] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:00.116493] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0b4, hex: EA03000000000000, int: 1002}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c0c4, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0d3, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0db, hex: D81187F92D2B0600, int: 1736326320165336}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0e3, hex: A0EFECA696AB1818, int: 1736326319885316000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0eb, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:00.116629] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=86][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:00.116671] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=15][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922106fe0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:136704} write_epoch=1, prev=this=0x2ad9220fa298 trans_version={val:1736325659490935000, v:0} scn={val:1736325659490935000, v:0} tx_id={txid:22448} prev=0x2ad9220f9c48 next=(nil) modify_count=4 acc_checksum=3997113610 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:34467} write_epoch=3, snapshot_version={val:1736326320073024000, v:0}, *this={this=0x2ad9220b7ec8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fa298 latest_compact_node=(nil) max_trans_version={val:1736325659490935000, v:0} max_trans_id=22448 max_elr_trans_version={val:1736325659490935000, v:0} max_elr_trans_id=22448 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325659490935000, v:0} min_modify_scn={val:1736324459170693000, v:0}}) [2025-01-08 00:52:00.116704] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=32][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922106fe0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:136704} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326349956382, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326349956382, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326320073024000, v:0}, tx_id:{txid:23718}, scn:{branch:0, seq:136701}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:23718}, tx_desc:{this:0x2ad8b86aa860, tx_id:{txid:23718}, state:1, addr:"192.168.159.138:2882", tenant_id:1001, session_id:3221815648, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326320106877, commit_ts:-1, finish_ts:-1, timeout_us:29999954, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326320088829, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6bd150, mem_ctx:0x2ad8fb6be6d0, tx_scn:{branch:0, seq:136704}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:00.116770] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=65][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:23718}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:136704}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:00.116783] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=13][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"BIGINT UNSIGNED":2},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:23718} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:00.116805] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=20][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:00.116813] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c174, hex: EA03000000000000, int: 1002}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c17c, hex: 0200000000000000, int: 2}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c184, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c193, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c19b, hex: D81187F92D2B0600, int: 1736326320165336}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c1a3, hex: 209BFF5E96AB1818, int: 1736326318678580000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c1ab, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:00.116873] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=38][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:00.116906] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922107178 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:136704} write_epoch=1, prev=this=0x2ad9220fa100 trans_version={val:1736325659490935000, v:0} scn={val:1736325659490935000, v:0} tx_id={txid:22448} prev=0x2ad9220f9ba0 next=(nil) modify_count=4 acc_checksum=1191713751 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:34467} write_epoch=3, snapshot_version={val:1736326320073024000, v:0}, *this={this=0x2ad9220b8080 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fa100 latest_compact_node=(nil) max_trans_version={val:1736325659490935000, v:0} max_trans_id=22448 max_elr_trans_version={val:1736325659490935000, v:0} max_elr_trans_id=22448 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325659490935000, v:0} min_modify_scn={val:1736324459170693000, v:0}}) [2025-01-08 00:52:00.117036] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=83][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922107178 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:136704} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326349956382, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326349956382, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326320073024000, v:0}, tx_id:{txid:23718}, scn:{branch:0, seq:136701}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:23718}, tx_desc:{this:0x2ad8b86aa860, tx_id:{txid:23718}, state:1, addr:"192.168.159.138:2882", tenant_id:1001, session_id:3221815648, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326320106877, commit_ts:-1, finish_ts:-1, timeout_us:29999954, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326320088829, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6bd150, mem_ctx:0x2ad8fb6be6d0, tx_scn:{branch:0, seq:136704}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:00.117109] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=119][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:23718}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:136704}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:00.117122] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=12][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:23718} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:00.117144] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=20][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:00.117153] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c234, hex: EA03000000000000, int: 1002}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c23c, hex: 0300000000000000, int: 3}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c244, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c253, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c25b, hex: D81187F92D2B0600, int: 1736326320165336}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c263, hex: 18B276A696AB1818, int: 1736326319877567000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c26b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:00.117261] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=84][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:00.117296] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922107310 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:136704} write_epoch=1, prev=this=0x2ad9220f9f68 trans_version={val:1736325659490935000, v:0} scn={val:1736325659490935000, v:0} tx_id={txid:22448} prev=0x2ad9220f9af8 next=(nil) modify_count=4 acc_checksum=3673667310 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:34467} write_epoch=3, snapshot_version={val:1736326320073024000, v:0}, *this={this=0x2ad9220b8238 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220f9f68 latest_compact_node=(nil) max_trans_version={val:1736325659490935000, v:0} max_trans_id=22448 max_elr_trans_version={val:1736325659490935000, v:0} max_elr_trans_id=22448 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325659490935000, v:0} min_modify_scn={val:1736324459170693000, v:0}}) [2025-01-08 00:52:00.117327] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=30][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922107310 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23718} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:136704} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326349956382, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326349956382, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326320073024000, v:0}, tx_id:{txid:23718}, scn:{branch:0, seq:136701}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:23718}, tx_desc:{this:0x2ad8b86aa860, tx_id:{txid:23718}, state:1, addr:"192.168.159.138:2882", tenant_id:1001, session_id:3221815648, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326320106877, commit_ts:-1, finish_ts:-1, timeout_us:29999954, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326320088829, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6bd150, mem_ctx:0x2ad8fb6be6d0, tx_scn:{branch:0, seq:136704}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:00.117391] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=62][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:23718}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:136704}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:00.117404] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=13][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"BIGINT UNSIGNED":4},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:23718} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:00.117424] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=20][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:00.117433] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c2f4, hex: EA03000000000000, int: 1002}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c2fc, hex: 0400000000000000, int: 4}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c304, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c313, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c31b, hex: D81187F92D2B0600, int: 1736326320165336}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c323, hex: FFFFFFFFFFFFFF3F, int: 4611686018427387903}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c32b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:00.117490] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=37][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:00.117555] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=10] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"192.168.159.138:42010"}) [2025-01-08 00:52:00.117663] WDIAG [STORAGE.TRANS] on_success (ob_trans_part_ctx.cpp:2253) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=48][errcode=-4389] transaction log sync use too much time(log_cb={ObTxBaseLogCb:{base_ts:{val:1736326319997056000, v:0}, log_ts:{val:1736326319997056000, v:0}, lsn:{lsn:84483377}, submit_ts:1736326319997772}, this:0x2ad89c0a3bd8, is_inited_:true, trans_id:{txid:28258}, ls_id:{id:1}, ctx:0x2ad89c0a01d0, tx_data_guard:{tx_data:NULL}, is_callbacked_:false, is_dynamic_:false, mds_range_:{range_array_.count():0, range_array_:[]}, cb_arg_array_:[{log_type:0x1, arg:null}, {log_type:0x40, arg:null}, {log_type:0x100, arg:null}], first_part_scn_:{val:18446744073709551615, v:3}, extra_cb_:null, need_free_extra_cb_:false, callbacks_.count():1}, log_sync_used_time=119888, ctx_lock_wait_time=1) [2025-01-08 00:52:00.125226] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=52] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:00.127109] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=98] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:00.129476] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=96] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:148371}, to_seq_no={branch:0, seq:136702}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:23718} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:00.129566] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=59] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:52:00.129579] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:152) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=12] transaction rpc statistics(total_rt=869, total_process=6, avg_rt=124) [2025-01-08 00:52:00.135301] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=98] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:00.135363] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=17] finish to resize log external storage handler(current_ts=1736326320135358, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:52:00.135386] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=21] snapshot_traffic_info once [2025-01-08 00:52:00.142120] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=85][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.146489] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326315701616000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:00.154706] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:52:00.154802] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1}, {id:1003}]) [2025-01-08 00:52:00.155307] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:00.155398] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=82] temporary file module use no blocks [2025-01-08 00:52:00.155413] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:00.155431] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:00.155450] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193824066396, flush_io_finished_ret_:0, flush_io_finished_round_:3806, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:00.155994] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=14] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"192.168.159.138:42022"}) [2025-01-08 00:52:00.160834] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:00.160860] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:00.161444] INFO [STORAGE.TRANS] rollback_to_savepoint (ob_trans_part_ctx.cpp:8738) [50624][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=92] rollback succeed trivially(trans_id={txid:23718}, ls_id={id:1}, op_sn=3, to_scn={branch:0, seq:136702}, last_scn={branch:0, seq:136702}) [2025-01-08 00:52:00.165346] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=94] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:00.165428] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=13] gc stale ls task succ [2025-01-08 00:52:00.173123] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185452-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:00.173524] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:00.173609] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=34] temporary file module use no blocks [2025-01-08 00:52:00.173630] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:00.173666] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:00.173692] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193824073146, flush_io_finished_ret_:0, flush_io_finished_round_:1978, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:00.179700] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=91][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84483907}], scn_array=[cnt:1, {val:1736326319997056001, v:0}], curr_size=122, accum_size=1253, time_cost=16091) [2025-01-08 00:52:00.179878] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43627167}], scn_array=[cnt:1, {val:1736326320157613000, v:0}], curr_size=750, accum_size=3731, time_cost=16458) [2025-01-08 00:52:00.184315] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58670][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:00.184449] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B87-0-0] [lt=102][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18400, log_task={header:{begin_lsn:{lsn:2239543}, end_lsn:{lsn:2239665}, log_id:18400, min_scn:{val:1736326319864970001, v:0}, max_scn:{val:1736326319864970001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2239430}, prev_proposal_id:1, committed_end_lsn:{lsn:2239543}, data_checksum:940009244, accum_checksum:651388072, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326320074795, freeze_ts:1736326320074795, submit_ts:1736326320074808, flushed_ts:1736326320077682, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:2874}, fs_cb_begin_ts=1736326320184437, log_life_time=109642) [2025-01-08 00:52:00.185177] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2218427}], scn_array=[cnt:1, {val:1736326319418178003, v:0}], curr_size=122, accum_size=356, time_cost=21723) [2025-01-08 00:52:00.185134] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] succ to submit log task to replay service(to_submit_lsn={lsn:2239599}, to_submit_scn={val:1736326319864970001, v:0}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326320184441, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2239665}, next_to_submit_scn_:{val:1736326319864970002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2239543}, end_lsn:{lsn:2239665}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2239665}, readable_log_tail:{lsn:2239665}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64865103, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326319864970001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326319864970001, v:0}, data_checksum:2124447694, flag:1149}}, init_mode_version:0, accumulate_checksum:651388072, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326319864970001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18310, miss_cnt:0, cache_read_size:2239665, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:00.185372] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2484) [46144][pnio1][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=23][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:23718}, sender_ls_id={id:1}, receiver_ls_id={id:9223372036854775807}, msg_type=67, status=-4018, receiver_addr="192.168.159.138:2882", request_id=1736324339343069) [2025-01-08 00:52:00.185398] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:288) [46144][pnio1][T0][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=25][errcode=0] trx rpc callback(ret=0, status=-4018, dst="192.168.159.138:2882", result={status:-4018, send_timestamp:1736326320161489, private_data:{val:18446744073709551615, v:3}}) [2025-01-08 00:52:00.188420] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:52:00.188506] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=157] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:52:00.189636] INFO pn_print_diag_info (group.c:755) [46142][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO client:0x2ad88e1634e8_"192.168.159.139:2882"_"192.168.159.139:2882"_132_1736322504706919_1, write_queue=0/0, write=70623/60879516, read=70620/15861940, doing=3, done=70620, write_time=85705761, read_time=63048, process_time=67715 [2025-01-08 00:52:00.189663] INFO pn_print_diag_info (group.c:755) [46142][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO client:0x2ad88e0ab4a8_"192.168.159.138:2882"_"192.168.159.138:2882"_129_1736322504704065_1, write_queue=0/0, write=70912/70348698, read=70911/16819699, doing=1, done=70911, write_time=86415553, read_time=127122, process_time=131955 [2025-01-08 00:52:00.189672] INFO pn_print_diag_info (group.c:755) [46142][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO client:0x2ad88e086048_"192.168.159.137:2882"_"192.168.159.137:2882"_125_1736322504683482_1, write_queue=0/0, write=17061/13880140, read=17061/4176322, doing=0, done=17062, write_time=19226342, read_time=1743, process_time=3552 [2025-01-08 00:52:00.189680] INFO pn_print_diag_info (group.c:768) [46142][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO server:0x2ad88e1fb468_"192.168.159.139:45062"_143_1736322505179614, write_queue=0/0, write=104420/59260624, read=104420/78074841, doing=0, done=104420, write_time=122399638, read_time=1405699, process_time=1414125 [2025-01-08 00:52:00.189689] INFO pn_print_diag_info (group.c:768) [46142][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO server:0x2ad88e1a9468_"192.168.159.138:42000"_139_1736322504928128, write_queue=0/0, write=107498/56200359, read=107498/82956405, doing=0, done=107498, write_time=125949482, read_time=706485, process_time=713660 [2025-01-08 00:52:00.189696] INFO pn_print_diag_info (group.c:768) [46142][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2ad88e0aa048_"192.168.159.137:41270"_127_1736322504689046, write_queue=0/0, write=17061/4176322, read=17061/13880140, doing=0, done=17061, write_time=18357640, read_time=2722, process_time=4235 [2025-01-08 00:52:00.189702] INFO pn_print_diag_info (group.c:775) [46142][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client_send:158597/145108693, client_queue_time=182744960, cnt=3, server_send:228979/119637305, server_queue_time=143343256, cnt=3 [2025-01-08 00:52:00.198680] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46493][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1959C6F8-0-0] [lt=19][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193825112510, cache_obj->added_lc()=false, cache_obj->get_object_id()=13698, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:00.211194] INFO pn_print_diag_info (group.c:755) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=93] PNIO client:0x2ad88e1fd4a8_"192.168.159.137:2882"_"192.168.159.137:2882"_147_1736322505625440_1, write_queue=0/0, write=16822/15634718, read=16822/4141853, doing=0, done=16822, write_time=18341657, read_time=1840, process_time=2982 [2025-01-08 00:52:00.211383] INFO pn_print_diag_info (group.c:755) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=184] PNIO client:0x2ad88e0aaa58_"192.168.159.139:2882"_"192.168.159.139:2882"_128_1736322504703688_1, write_queue=0/0, write=70605/68129551, read=70602/15751163, doing=3, done=70602, write_time=86483356, read_time=36234, process_time=42000 [2025-01-08 00:52:00.211399] INFO pn_print_diag_info (group.c:755) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO client:0x2ad88e0874e8_"192.168.159.138:2882"_"192.168.159.138:2882"_126_1736322504683741_1, write_queue=0/0, write=71170/57846926, read=71169/15759467, doing=1, done=71170, write_time=87354950, read_time=14291, process_time=19513 [2025-01-08 00:52:00.211411] INFO pn_print_diag_info (group.c:768) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO server:0x2ad8ae186048_"192.168.159.139:45082"_151_1736322506111955, write_queue=0/0, write=104726/61989158, read=104726/78882302, doing=0, done=104726, write_time=122742143, read_time=947147, process_time=955601 [2025-01-08 00:52:00.211423] INFO pn_print_diag_info (group.c:768) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO server:0x2ad88e1ff4a8_"192.168.159.138:42022"_150_1736322505867001, write_queue=1/806, write=107312/67711191, read=107313/83600409, doing=1, done=107312, write_time=126179881, read_time=703967, process_time=712033 [2025-01-08 00:52:00.211430] INFO pn_print_diag_info (group.c:768) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO server:0x2ad88e1fea98_"192.168.159.137:41292"_149_1736322505627594, write_queue=0/0, write=16822/4141853, read=16822/15634718, doing=0, done=16822, write_time=18245008, read_time=3199, process_time=4739 [2025-01-08 00:52:00.211437] INFO pn_print_diag_info (group.c:775) [46143][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client_send:158598/141611534, client_queue_time=168814884, cnt=3, server_send:228861/133843008, server_queue_time=138256024, cnt=3 [2025-01-08 00:52:00.212084] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D89-0-0] [lt=36] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1961) [2025-01-08 00:52:00.212180] INFO [LIB] stat (utility.h:1210) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D89-0-0] [lt=28] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:00.212198] INFO [LIB] stat (utility.h:1210) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D89-0-0] [lt=17] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=301388, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:00.212680] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=21] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8835, offset=2510233}, end_cursor:ObLogCursor{file_id=1, log_id=8836, offset=2510530}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d04050, len:615, log_data_len:297, seq:8835, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:00.214606] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB621-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:00.214645] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB621-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326319864970002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326319864970002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:00.216782] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8836, offset=2510530}, end_cursor:ObLogCursor{file_id=1, log_id=8837, offset=2510827}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d08050, len:4414, log_data_len:297, seq:8836, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:00.220402] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=78, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:00.220459] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:00.222413] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58670][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A96B5-0-0] [lt=140] init point thread id with(&point=0x5572d50e74c0, idx_=1326, point=[thread id=58670, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58670) [2025-01-08 00:52:00.228723] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=88907, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:00.228775] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=1757, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:00.228860] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=6] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=30, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:00.243059] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.250757] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=44] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6714388480, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:00.250857] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:52:00.272397] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [58653][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=58] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:00.275893] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=50] ====== check clog disk timer task ====== [2025-01-08 00:52:00.276014] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=96] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:52:00.283152] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4015] clock out of order(cur_ts=1736326320283147, cur_ts_=1736326320251443, delta=31704) [2025-01-08 00:52:00.292859] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A85-0-0] [lt=15] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326320292769}) [2025-01-08 00:52:00.293138] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B9-0-0] [lt=25] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:00.305594] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=25] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"192.168.159.138:42000"}) [2025-01-08 00:52:00.307155] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=73][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2218549}], scn_array=[cnt:1, {val:1736326319418178004, v:0}], curr_size=122, accum_size=478, time_cost=38380) [2025-01-08 00:52:00.307392] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=75] PNIO [ratelimit] time: 1736326320307387, bytes: 0, bw: 0.000000 MB/s, add_ts: 1003397, add_bytes: 0 [2025-01-08 00:52:00.307432] INFO pn_print_diag_info (group.c:775) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=38] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-01-08 00:52:00.308965] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0CE-0-0] [lt=53][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:00.309717] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=62][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18740523}], scn_array=[cnt:1, {val:1736326319418178004, v:0}], curr_size=122, accum_size=967, time_cost=40912) [2025-01-08 00:52:00.309798] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43628506}], scn_array=[cnt:1, {val:1736326320251443000, v:0}], curr_size=589, accum_size=4909, time_cost=45836) [2025-01-08 00:52:00.312264] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87193-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:00.312316] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:00.312328] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:00.313731] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58615][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48E9-0-0] [lt=14][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18504, log_task={header:{begin_lsn:{lsn:2252204}, end_lsn:{lsn:2252326}, log_id:18504, min_scn:{val:1736326319877567000, v:0}, max_scn:{val:1736326319877567000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2252082}, prev_proposal_id:1, committed_end_lsn:{lsn:2252204}, data_checksum:1715074492, accum_checksum:394973942, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326319964173, freeze_ts:1736326319964173, submit_ts:1736326319964185, flushed_ts:1736326319998499, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:34314}, fs_cb_begin_ts=1736326320313719, log_life_time=349546) [2025-01-08 00:52:00.320148] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B9-0-0] [lt=29] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326320320115, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326320320115, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326320320115, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:00.320267] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859B9-0-0] [lt=117] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326320292769}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326320320115, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326320320115, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326320320115, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:00.320788] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.320826] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.325465] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA19-0-0] [lt=52] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"192.168.159.138:41998"}) [2025-01-08 00:52:00.328160] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:00.340333] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=111] get number(ret=0, service_type_=0, range=1, base_id=1736326320330736000, start_id=1736326320330736000, end_id=1736326320330736001) [2025-01-08 00:52:00.340440] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=56] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:00.344062] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [58658][T1_TimerWK5][T1][Y0-0000000000000000-0-0] [lt=43] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=16637952) [2025-01-08 00:52:00.345241] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.346552] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50915][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F5F7-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326320344553000, start_id=1736326320344553000, end_id=1736326320344553001) [2025-01-08 00:52:00.357835] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:52:00.358149] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58660][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=36] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:52:00.372226] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=5, avg_log_batch_cnt=1, total_group_log_size=1089, avg_group_log_size=217) [2025-01-08 00:52:00.372281] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1089) [2025-01-08 00:52:00.372785] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:00.372866] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4915, iops:5, bw:22728, [delay/us]:prepare:7, schedule:456, submit:113, rt:105044, total:105622) [2025-01-08 00:52:00.372898] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=1, iops=1, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:00.373002] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=87] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4915, avg_iops:5, avg_bw:22729, [delay/us]: prepare:7, schedule:456, submit:113, device:105044, total:105622) [2025-01-08 00:52:00.373025] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:00.373267] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:8, bw:37881, [delay/us]:prepare:7, schedule:4137, submit:1497, rt:73619, total:79262) [2025-01-08 00:52:00.373313] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:00.373557] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=52] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:37873, [delay/us]: prepare:7, schedule:4137, submit:1497, device:73619, total:79262) [2025-01-08 00:52:00.373673] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=95] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:00.373712] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:18935, [delay/us]:prepare:9, schedule:2573, submit:1, rt:19778, total:22362) [2025-01-08 00:52:00.373781] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:3, hold_mem:24576 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:16, bw:64382, [delay/us]:prepare:6, schedule:2539, submit:287, rt:82159, total:84992) [2025-01-08 00:52:00.373819] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS TENANT](tenant_id=1002, ref_cnt=8, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49997, free_result_cnt=49997, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:00.373970] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=136] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:18937, [delay/us]: prepare:9, schedule:2573, submit:1, device:19778, total:22362) [2025-01-08 00:52:00.373989] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:16, avg_bw:64386, [delay/us]: prepare:6, schedule:2539, submit:287, device:82159, total:84992) [2025-01-08 00:52:00.374027] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:00.374050] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:3787, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:547433, total:547441) [2025-01-08 00:52:00.374165] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:00.374325] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=152] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3787, [delay/us]: prepare:0, schedule:0, submit:600000000, device:547433, total:547441) [2025-01-08 00:52:00.374354] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:00.374397] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:00.374418] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:00.374473] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=52] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:00.374482] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=219kB/s, net_out=135kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:00.375375] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=36] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:00.376037] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=6] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:00.377255] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2252448}], scn_array=[cnt:1, {val:1736326320344553000, v:0}], curr_size=122, accum_size=366, time_cost=13390) [2025-01-08 00:52:00.381400] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA1A-0-0] [lt=8] get number(ret=0, service_type_=0, range=1, base_id=1736326320379116000, start_id=1736326320379116000, end_id=1736326320379116001) [2025-01-08 00:52:00.381691] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1EA9BA1A-0-0] [lt=58] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5498) [2025-01-08 00:52:00.386935] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2239909}], scn_array=[cnt:1, {val:1736326320344553001, v:0}], curr_size=122, accum_size=488, time_cost=15128) [2025-01-08 00:52:00.388823] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=38] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:00.390850] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:52:00.398407] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=40] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:172K/39:2303583 disk:172K/39:2303583 t1m2:28K/6:375190 t1m2g0:28K/6:375190 t1002m2:96K/23:2194362 t1002m2g0:96K/23:2194362 t1001m2:48K/10:3707212 t1001m2g0:48K/10:3707212 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:00.410761] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43631546}], scn_array=[cnt:1, {val:1736326320398093000, v:0}], curr_size=750, accum_size=8110, time_cost=10546) [2025-01-08 00:52:00.424254] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2082) [2025-01-08 00:52:00.434023] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58671][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:00.434165] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:52:00.437661] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58672][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:00.437723] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95] worker thread created(tenant_->id()=1001, group_id_=1, token=8) [2025-01-08 00:52:00.446367] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=160][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.447345] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=211] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5321416607, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:00.447653] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=66] recycle one thread(this=0x2ad8885abcf0, total=5, remain=4) [2025-01-08 00:52:00.447721] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=67] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5321416607, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:00.453286] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=46][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efe0a030, session_id:3221587633, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1106, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:00.453347] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=52] kill and revert session(conn.sessid_=3221587633, proxy_sessid=0, ret=0) [2025-01-08 00:52:00.453369] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] sock ref clean, do destroy(*s={this:0x2ad8efe0a030, session_id:3221587633, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326317192724, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1106, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:00.453408] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] unlock session of tenant(conn.sessid_=3221587633, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:00.453430] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] connection close(sessid=3221587633, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CEB-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:1001, rec_:1001, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:5, seq_:7}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:37, seq_:45}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:46}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:00.460621] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:00.460705] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=80] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:52:00.460878] INFO [SERVER] run (obmp_disconnect.cpp:83) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=46] free session successfully(sessid=3221587633) [2025-01-08 00:52:00.460898] INFO [SERVER] run (obmp_disconnect.cpp:90) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=19] mark session id unused(sessid=3221587633) [2025-01-08 00:52:00.463563] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.463650] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.495883] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=45] ls blacklist refresh finish(cost_time=205438) [2025-01-08 00:52:00.520866] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.542454] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.544560] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.547875] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.550418] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=204] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:52:00.594816] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=61][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2240153}], scn_array=[cnt:1, {val:1736326320344553003, v:0}], curr_size=122, accum_size=732, time_cost=21625) [2025-01-08 00:52:00.595341] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84485322}], scn_array=[cnt:1, {val:1736326320419312001, v:0}], curr_size=122, accum_size=2668, time_cost=15875) [2025-01-08 00:52:00.605532] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18740889}], scn_array=[cnt:1, {val:1736326320330736002, v:0}], curr_size=122, accum_size=1333, time_cost=22367) [2025-01-08 00:52:00.607008] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=69][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43633635}], scn_array=[cnt:1, {val:1736326320593324000, v:0}], curr_size=589, accum_size=10038, time_cost=12392) [2025-01-08 00:52:00.635722] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84485444}], scn_array=[cnt:1, {val:1736326320622896000, v:0}], curr_size=586, accum_size=3254, time_cost=11377) [2025-01-08 00:52:00.648480] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.665027] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:00.665108] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:52:00.665831] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=45] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:52:00.665871] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:52:00.665894] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=19] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=92) [2025-01-08 00:52:00.672575] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1736326320672570, bytes: 885255315, bw: 0.319779 MB/s, add_ts: 1001241, add_bytes: 335729 [2025-01-08 00:52:00.672642] INFO pn_print_diag_info (group.c:755) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=60] PNIO client:0x2ad88e1fe048_"192.168.159.139:2882"_"192.168.159.139:2882"_148_1736322505625884_1, write_queue=0/0, write=70779/70966059, read=70779/15667046, doing=0, done=70779, write_time=91106656, read_time=15128, process_time=21185 [2025-01-08 00:52:00.672717] INFO pn_print_diag_info (group.c:755) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO client:0x2ad88e162a98_"192.168.159.138:2882"_"192.168.159.138:2882"_131_1736322504706708_1, write_queue=0/0, write=70898/61084870, read=70896/15798151, doing=0, done=70899, write_time=87433687, read_time=48564, process_time=55853 [2025-01-08 00:52:00.672744] INFO pn_print_diag_info (group.c:755) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO client:0x2ad88e004808_"192.168.159.137:2882"_"192.168.159.137:2882"_122_1736322504678620_1, write_queue=0/0, write=16948/15687384, read=16948/4187397, doing=0, done=16950, write_time=19097072, read_time=2094, process_time=3200 [2025-01-08 00:52:00.672804] INFO pn_print_diag_info (group.c:768) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=57] PNIO server:0x2ad88e1faa58_"192.168.159.139:45058"_142_1736322505176841, write_queue=0/0, write=104460/48201283, read=104460/79258326, doing=0, done=104460, write_time=131654899, read_time=933028, process_time=941516 [2025-01-08 00:52:00.672817] INFO pn_print_diag_info (group.c:768) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO server:0x2ad88e1a8a58_"192.168.159.138:41998"_138_1736322504924846, write_queue=0/0, write=107354/53026844, read=107354/82844248, doing=0, done=107354, write_time=132657258, read_time=784339, process_time=793041 [2025-01-08 00:52:00.672825] INFO pn_print_diag_info (group.c:768) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO server:0x2ad88e005258_"192.168.159.137:41266"_123_1736322504679850, write_queue=0/0, write=16948/4187397, read=16948/15687384, doing=0, done=16948, write_time=18257706, read_time=3382, process_time=4525 [2025-01-08 00:52:00.672832] INFO pn_print_diag_info (group.c:775) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client_send:158628/147739403, client_queue_time=180907643, cnt=3, server_send:228762/105415524, server_queue_time=156969821, cnt=3 [2025-01-08 00:52:00.686540] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D2248464E-0-0] [lt=100] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:52:00.696841] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [46340][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=33] XA scheduler heartbeat task statistics(avg_time=0) [2025-01-08 00:52:00.701054] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=37] current monitor number(seq_=-1) [2025-01-08 00:52:00.703406] INFO pn_print_diag_info (group.c:755) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO client:0x2ad88e1fca58_"192.168.159.138:2882"_"192.168.159.138:2882"_146_1736322505625436_1, write_queue=0/0, write=71009/72616141, read=71009/16246583, doing=0, done=71010, write_time=92885289, read_time=104577, process_time=110734 [2025-01-08 00:52:00.703445] INFO pn_print_diag_info (group.c:755) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=35] PNIO client:0x2ad88e162048_"192.168.159.137:2882"_"192.168.159.137:2882"_130_1736322504706680_1, write_queue=0/0, write=16728/16758290, read=16727/4108643, doing=0, done=16728, write_time=18243638, read_time=1764, process_time=3130 [2025-01-08 00:52:00.703456] INFO pn_print_diag_info (group.c:755) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO client:0x2ad88e086a98_"192.168.159.139:2882"_"192.168.159.139:2882"_124_1736322504683494_1, write_queue=0/0, write=70890/61580527, read=70890/17045599, doing=0, done=70891, write_time=89211158, read_time=232184, process_time=238568 [2025-01-08 00:52:00.703464] INFO pn_print_diag_info (group.c:768) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2ad88e1fc048_"192.168.159.139:45074"_144_1736322505188012, write_queue=0/0, write=104472/54718906, read=104472/109491433, doing=0, done=104472, write_time=125925182, read_time=2771804, process_time=2779952 [2025-01-08 00:52:00.703471] INFO pn_print_diag_info (group.c:768) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO server:0x2ad88e1fa048_"192.168.159.138:42010"_140_1736322504947415, write_queue=0/0, write=107537/53453208, read=107537/82553580, doing=0, done=107537, write_time=126566831, read_time=1197049, process_time=1205663 [2025-01-08 00:52:00.703478] INFO pn_print_diag_info (group.c:768) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x2ad88e1a8048_"192.168.159.137:41278"_133_1736322504708541, write_queue=0/0, write=16727/4108643, read=16728/16758290, doing=0, done=16728, write_time=18015911, read_time=3741, process_time=5028 [2025-01-08 00:52:00.703483] INFO pn_print_diag_info (group.c:775) [46144][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO client_send:158629/150955636, client_queue_time=199230379, cnt=3, server_send:228737/112280979, server_queue_time=151608570, cnt=3 [2025-01-08 00:52:00.712576] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884F01-0-0] [lt=107] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:00.745125] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884F02-0-0] [lt=48] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:00.749023] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-01-08 00:52:00.749184] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=109][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.757828] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884F03-0-0] [lt=37] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:00.761967] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=48] success to update min reserved snapshot(reserved_snapshot=1736324514412287000, duration=1800, snapshot_gc_ts=1736326314412287000) [2025-01-08 00:52:00.762511] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=38] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:52:00.762591] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=71] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:52:00.762608] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:52:00.762616] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:52:00.762623] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:52:00.762630] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=95) [2025-01-08 00:52:00.762980] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=24] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:52:00.763022] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=37] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:52:00.763036] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=12] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:52:00.763045] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=8] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:00.765716] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:52:00.766821] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=1100] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:00.774414] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=51] update reserved snapshot finished(cost_ts=12449, reserved_snapshot=1736324514412287000) [2025-01-08 00:52:00.779595] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884F04-0-0] [lt=31] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:00.782701] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=28] failed to get result(ret=-4008) [2025-01-08 00:52:00.782889] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=95] check new jobs(ret=0, tenant_id=1, is_oracle_tenant=false, job_infos=[{tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326178662155, start_date_:1736323478339648, last_date_:1736326178341674, this_date_:0, next_date_:1736327078339648, end_date_:64060560000000000, total_:8578451, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736408978339542, last_date_:0, this_date_:0, next_date_:1736408978339542, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736322578351862, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:52:00.783080] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [46356][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=188] check all tenants(ret=0) [2025-01-08 00:52:00.786613] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085179-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:00.787178] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:52:00.799584] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884F06-0-0] [lt=9] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:00.801774] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884F07-0-0] [lt=43] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:00.814397] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=99][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43634924}], scn_array=[cnt:1, {val:1736326320799644000, v:0}], curr_size=588, accum_size=11326, time_cost=14390) [2025-01-08 00:52:00.825912] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884F08-0-0] [lt=40] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:00.829771] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:52:00.829942] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=95][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:52:00.830064] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=26] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-01-08 00:52:00.831787] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=47] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:00.831993] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58671][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A28-0-0] [lt=107] init point thread id with(&point=0x5572d50e7500, idx_=1327, point=[thread id=58671, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58671) [2025-01-08 00:52:00.838637] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D1EEBAD75-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1736326320837766000, start_id=1736326320837766000, end_id=1736326320837766001) [2025-01-08 00:52:00.848410] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15278065393, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:00.848504] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:00.848521] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x5572d527f900, thread=0x2ad8bf5d5c90) [2025-01-08 00:52:00.848531] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15278065393, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:00.848543] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15209311293, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:52:00.848553] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:00.848560] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6] try_wait return eagain(this=0x2ad8881fc3b0, thread=0x2ad8aedd5ec0) [2025-01-08 00:52:00.848565] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15209311293, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:52:00.848574] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:8007224381, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:00.848584] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:00.848590] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6] try_wait return eagain(this=0x2ad8c0eb0e70, thread=0x2ad934893970) [2025-01-08 00:52:00.848596] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=5] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:8007224381, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:00.849659] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=95][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.852946] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:00.853355] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=316] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:00.853828] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=29] gc stale ls task succ [2025-01-08 00:52:00.853867] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:00.853887] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-01-08 00:52:00.853896] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:00.853910] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:00.853923] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193824766598, flush_io_finished_ret_:0, flush_io_finished_round_:1983, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:00.862386] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDE-0-0] [lt=56] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:00.862431] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDE-0-0] [lt=41] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:00.863120] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=29][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922108048 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43665} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:11785} write_epoch=1, prev=this=0x2ad9220fb1f8 trans_version={val:1736325660857250000, v:0} scn={val:1736325660857250000, v:0} tx_id={txid:42404} prev=0x2ad9220faba8 next=(nil) modify_count=4 acc_checksum=2463213461 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:21417} write_epoch=3, snapshot_version={val:1736326320854470000, v:0}, *this={this=0x2ad9220b1688 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fb1f8 latest_compact_node=(nil) max_trans_version={val:1736325660857250000, v:0} max_trans_id=42404 max_elr_trans_version={val:1736325660857250000, v:0} max_elr_trans_id=42404 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325660857250000, v:0} min_modify_scn={val:1736324400322864000, v:0}}) [2025-01-08 00:52:00.863177] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=55][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922108048 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43665} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:11785} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326350760191, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326350760191, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326320854470000, v:0}, tx_id:{txid:43665}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:43665}, tx_desc:{this:0x2ad8b86d5040, tx_id:{txid:43665}, state:1, addr:"192.168.159.139:2882", tenant_id:1001, session_id:3222077492, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326320862332, commit_ts:-1, finish_ts:-1, timeout_us:29999937, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326320948785, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6b4a50, mem_ctx:0x2ad8fb6b5fd0, tx_scn:{branch:0, seq:11785}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:00.863336] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=157][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:43665}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:11785}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:00.863354] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"BIGINT UNSIGNED":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:43665} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:00.863387] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:00.863399] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0b4, hex: EA03000000000000, int: 1002}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c0c4, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0d3, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0db, hex: DFF591F92D2B0600, int: 1736326320879071}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0e3, hex: 702BB2DF96AB1818, int: 1736326320837766000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0eb, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:00.863537] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=96][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:00.863576] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=13][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad9221081e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43665} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:11785} write_epoch=1, prev=this=0x2ad9220fb060 trans_version={val:1736325660857250000, v:0} scn={val:1736325660857250000, v:0} tx_id={txid:42404} prev=0x2ad9220fab00 next=(nil) modify_count=4 acc_checksum=616429942 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:21417} write_epoch=3, snapshot_version={val:1736326320854470000, v:0}, *this={this=0x2ad9220b1840 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fb060 latest_compact_node=(nil) max_trans_version={val:1736325660857250000, v:0} max_trans_id=42404 max_elr_trans_version={val:1736325660857250000, v:0} max_elr_trans_id=42404 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325660857250000, v:0} min_modify_scn={val:1736324400322864000, v:0}}) [2025-01-08 00:52:00.863670] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=92][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad9221081e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43665} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:11785} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326350760191, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326350760191, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326320854470000, v:0}, tx_id:{txid:43665}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:43665}, tx_desc:{this:0x2ad8b86d5040, tx_id:{txid:43665}, state:1, addr:"192.168.159.139:2882", tenant_id:1001, session_id:3222077492, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326320862332, commit_ts:-1, finish_ts:-1, timeout_us:29999937, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326320948785, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6b4a50, mem_ctx:0x2ad8fb6b5fd0, tx_scn:{branch:0, seq:11785}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:00.863861] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=189][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:43665}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:11785}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:00.863877] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"BIGINT UNSIGNED":2},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:43665} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:00.863900] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=21][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:00.863909] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c174, hex: EA03000000000000, int: 1002}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c17c, hex: 0200000000000000, int: 2}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c184, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c193, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c19b, hex: DFF591F92D2B0600, int: 1736326320879071}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c1a3, hex: 818179C196AB1818, int: 1736326320330736001}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c1ab, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:00.863972] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=40][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:00.864007] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922108378 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43665} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:11785} write_epoch=1, prev=this=0x2ad9220faec8 trans_version={val:1736325660857250000, v:0} scn={val:1736325660857250000, v:0} tx_id={txid:42404} prev=0x2ad9220faa58 next=(nil) modify_count=4 acc_checksum=2180437195 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:21417} write_epoch=3, snapshot_version={val:1736326320854470000, v:0}, *this={this=0x2ad9220b19f8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220faec8 latest_compact_node=(nil) max_trans_version={val:1736325660857250000, v:0} max_trans_id=42404 max_elr_trans_version={val:1736325660857250000, v:0} max_elr_trans_id=42404 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325660857250000, v:0} min_modify_scn={val:1736324400322864000, v:0}}) [2025-01-08 00:52:00.864038] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=31][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922108378 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43665} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:11785} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326350760191, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326350760191, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326320854470000, v:0}, tx_id:{txid:43665}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:43665}, tx_desc:{this:0x2ad8b86d5040, tx_id:{txid:43665}, state:1, addr:"192.168.159.139:2882", tenant_id:1001, session_id:3222077492, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326320862332, commit_ts:-1, finish_ts:-1, timeout_us:29999937, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326320948785, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6b4a50, mem_ctx:0x2ad8fb6b5fd0, tx_scn:{branch:0, seq:11785}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:00.864102] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=63][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:43665}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:11785}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:00.864115] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=12][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:43665} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:00.864136] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=20][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:00.864144] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c234, hex: EA03000000000000, int: 1002}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c23c, hex: 0300000000000000, int: 3}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c244, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c253, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c25b, hex: DFF591F92D2B0600, int: 1736326320879071}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c263, hex: 29564CC296AB1818, int: 1736326320344553001}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c26b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:00.864263] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=94][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:00.864299] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=12][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922108510 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43665} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:11785} write_epoch=1, prev=this=0x2ad9220fad30 trans_version={val:1736325660857250000, v:0} scn={val:1736325660857250000, v:0} tx_id={txid:42404} prev=0x2ad9220fa9b0 next=(nil) modify_count=4 acc_checksum=1098086445 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:21417} write_epoch=3, snapshot_version={val:1736326320854470000, v:0}, *this={this=0x2ad9220b1bb0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fad30 latest_compact_node=(nil) max_trans_version={val:1736325660857250000, v:0} max_trans_id=42404 max_elr_trans_version={val:1736325660857250000, v:0} max_elr_trans_id=42404 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325660857250000, v:0} min_modify_scn={val:1736324400322864000, v:0}}) [2025-01-08 00:52:00.864390] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=89][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922108510 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43665} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:11785} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326350760191, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326350760191, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326320854470000, v:0}, tx_id:{txid:43665}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:43665}, tx_desc:{this:0x2ad8b86d5040, tx_id:{txid:43665}, state:1, addr:"192.168.159.139:2882", tenant_id:1001, session_id:3222077492, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326320862332, commit_ts:-1, finish_ts:-1, timeout_us:29999937, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326320948785, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6b4a50, mem_ctx:0x2ad8fb6b5fd0, tx_scn:{branch:0, seq:11785}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:00.864598] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=206][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:43665}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:11785}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:00.864615] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"BIGINT UNSIGNED":4},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:43665} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:00.864693] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=76][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:00.864703] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c2f4, hex: EA03000000000000, int: 1002}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c2fc, hex: 0400000000000000, int: 4}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c304, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c313, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c31b, hex: DFF591F92D2B0600, int: 1736326320879071}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c323, hex: FFFFFFFFFFFFFF3F, int: 4611686018427387903}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c32b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:00.864813] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=86][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:00.864906] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=11] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"192.168.159.139:45074"}) [2025-01-08 00:52:00.865017] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:00.865446] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48F5-0-0] [lt=49][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18510, log_task={header:{begin_lsn:{lsn:2252936}, end_lsn:{lsn:2253058}, log_id:18510, min_scn:{val:1736326320344553004, v:0}, max_scn:{val:1736326320344553004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2252814}, prev_proposal_id:1, committed_end_lsn:{lsn:2252936}, data_checksum:1715074492, accum_checksum:1773787656, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326320761610, freeze_ts:1736326320761610, submit_ts:1736326320761623, flushed_ts:1736326320771863, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:10240}, fs_cb_begin_ts=1736326320865435, log_life_time=103825) [2025-01-08 00:52:00.867313] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884F09-0-0] [lt=31] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:00.869481] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58670][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B95-0-0] [lt=52][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18407, log_task={header:{begin_lsn:{lsn:2240397}, end_lsn:{lsn:2240519}, log_id:18407, min_scn:{val:1736326320344553005, v:0}, max_scn:{val:1736326320344553005, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2240275}, prev_proposal_id:1, committed_end_lsn:{lsn:2240397}, data_checksum:940009244, accum_checksum:3504964585, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326320767722, freeze_ts:1736326320767722, submit_ts:1736326320767729, flushed_ts:1736326320776149, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:8420}, fs_cb_begin_ts=1736326320869418, log_life_time=101696) [2025-01-08 00:52:00.869563] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [46356][T1_DBMSSched][T1][YB42C0A89F89-00062B2D1A884F0A-0-0] [lt=23] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:00.870659] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=57] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:00.870767] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:52:00.877151] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=28] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:35283}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:43665} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:00.877797] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC437-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:00.878051] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:00.878309] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:00.881333] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=225] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:00.888496] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=160] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13634, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:00.888563] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1210) [2025-01-08 00:52:00.888588] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1388, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:00.888622] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1210) [2025-01-08 00:52:00.888643] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:00.890622] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1210, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:00.890716] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11033, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:00.890851] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=133] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1699) [2025-01-08 00:52:00.890869] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3210, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:00.891289] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:00.891346] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1699) [2025-01-08 00:52:00.891364] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=35, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:00.893136] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326320463973000, v:0}, new_max_version={val:1736326320463973000, v:0}, last_min_version={val:1736326320398093000, v:0}, last_max_version={val:1736326320398093000, v:0}, total_time_us=30239, delta_us=429161) [2025-01-08 00:52:00.893255] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=114] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326320463973000, v:0}, min_version={val:1736326320463973000, v:0}, max_version={val:1736326320463973000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:00.894408] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:00.894708] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2240697}, scn={val:1736326320841974001, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:52:00.894746] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:25, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326320891653, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2240753}, next_to_submit_scn_:{val:1736326320841974002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2240641}, end_lsn:{lsn:2240753}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2240753}, readable_log_tail:{lsn:2240753}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64864015, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326320841974001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326320841974001, v:0}, data_checksum:1205275101, flag:45748}}, init_mode_version:0, accumulate_checksum:419639111, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326320841974001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18319, miss_cnt:0, cache_read_size:2240753, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2240697}, scn_:{val:1736326320841974001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326320911279, first_handle_ts_:1736326320894745, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad96dc04150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:52:00.895588] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=87] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324515701616000) [2025-01-08 00:52:00.895616] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2240697}, scn_:{val:1736326320841974001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326320911279, first_handle_ts_:1736326320895570, replay_cost_:43, retry_cost_:-1, read_log_buf_:0x2ad96dc04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326320891653, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2240753}, next_to_submit_scn_:{val:1736326320841974002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2240641}, end_lsn:{lsn:2240753}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2240753}, readable_log_tail:{lsn:2240753}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64864015, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326320841974001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326320841974001, v:0}, data_checksum:1205275101, flag:45748}}, init_mode_version:0, accumulate_checksum:419639111, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326320841974001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18319, miss_cnt:0, cache_read_size:2240753, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=15, ret=0) [2025-01-08 00:52:00.898685] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.898809] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.899104] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.899200] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:00.900547] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD77-0-0] [lt=22] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"192.168.159.139:45058"}) [2025-01-08 00:52:00.902780] INFO [LIB] stat (utility.h:1210) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD78-0-0] [lt=110] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=84, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:00.902819] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1EEBAD78-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=13657, avg_group_log_size=758) [2025-01-08 00:52:00.904426] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=63] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=12005, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:00.904509] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=81] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=13779) [2025-01-08 00:52:00.904536] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=1310, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:00.905280] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=12504) [2025-01-08 00:52:00.905309] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=63, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:00.919902] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=53] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18741377}, readable_log_tail:{lsn:18741377}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48363391, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:00.920056] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=152] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:52:00.920076] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=16] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2219403}, readable_log_tail:{lsn:2219403}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64885365, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:00.920257] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=179] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:52:00.920306] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=42] get_replay_process success(id={id:1002}, submitted_log_size=2253180, unsubmitted_log_size=2253180, replayed_log_size=2253180, unreplayed_log_size=0) [2025-01-08 00:52:00.920320] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] get_replay_process success(id={id:1003}, submitted_log_size=2240753, unsubmitted_log_size=2240753, replayed_log_size=2240753, unreplayed_log_size=0) [2025-01-08 00:52:00.920349] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:52:00.933821] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC6-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:00.936341] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC6-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:00.949965] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:00.961825] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD95-0-0] [lt=8] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326321071437, keepalive_ts:-1}) [2025-01-08 00:52:00.961942] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD95-0-0] [lt=115] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326321071437, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:00.963404] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48F7-0-0] [lt=45] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:00.963471] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48F7-0-0] [lt=51] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:52:00.963491] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48F7-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:52:00.964605] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=100] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=8302, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:00.964701] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=93] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:52:00.964744] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=849, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:00.964865] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=64, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:00.969863] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B97-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1323) [2025-01-08 00:52:00.973941] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1445, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:00.973910] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB15-0-0] [lt=41] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=32, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:00.973984] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB15-0-0] [lt=72] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=49568, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:00.973999] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB15-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=9485, avg_log_submit_to_slide_time=49546) [2025-01-08 00:52:00.973986] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18320, miss_cnt:0, cache_read_size:2240875, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:00.974077] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=103] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=351, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:00.974332] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=254] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:00.981439] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=82, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:00.981566] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=47, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:00.981656] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1210, avg_group_log_size=121) [2025-01-08 00:52:00.981679] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1210) [2025-01-08 00:52:00.981694] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:00.982166] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=480] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3539, avg_group_log_size=353) [2025-01-08 00:52:00.983798] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4084) [2025-01-08 00:52:00.986862] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D91-0-0] [lt=20] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=85, avg_log_submit_to_flush_time=50260, avg_log_submit_to_slide_time=89557) [2025-01-08 00:52:00.986977] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=324] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4206, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:00.987144] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A96-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:00.987165] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A96-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38054, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:00.987244] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A96-0-0] [lt=77] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=4305, avg_log_gen_to_submit_time=4417, avg_log_submit_to_flush_time=13985, avg_log_submit_to_slide_time=33636) [2025-01-08 00:52:00.987337] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1821, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:00.987711] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A97-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=27, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:00.987725] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A97-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38829, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:01.005825] INFO [LIB] stat (utility.h:1210) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F6A-0-0] [lt=31] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=15, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:01.005860] INFO [LIB] stat (utility.h:1210) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F6A-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=41814, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:01.005872] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58604][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F6A-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=20, avg_log_gen_to_freeze_time=4644, avg_log_gen_to_submit_time=4742, avg_log_submit_to_flush_time=14043, avg_log_submit_to_slide_time=37071) [2025-01-08 00:52:01.006139] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=14368, hit_cnt=20, read_cnt=20, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:01.024479] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D1969108A-0-0] [lt=29] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.784775571823120117e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=892, Throttle Trigger(MB)=535, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.806757882735221798e+01) [2025-01-08 00:52:01.027658] INFO [SERVER] refresh_avail_memory (ob_table_load_assigned_memory_manager.cpp:117) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=123] ObTableLoadAssignedMemoryManager::refresh_avail_memory(MTL_ID()=1002, avail_memory=161061273, chunk_count_=0, avail_sort_memory_=161061273, avail_memory_=161061273) [2025-01-08 00:52:01.029906] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:01.050129] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.061495] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48F9-0-0] [lt=7] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=19, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:01.061543] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48F9-0-0] [lt=47] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=84522, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:01.061572] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:01.061570] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58561][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A48F9-0-0] [lt=25] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=18, avg_log_submit_to_flush_time=9378, avg_log_submit_to_slide_time=84503) [2025-01-08 00:52:01.061615] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18419, miss_cnt:0, cache_read_size:2253302, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:01.069419] INFO [LIB] stat (utility.h:1210) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B99-0-0] [lt=56] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:01.069500] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58561][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3B99-0-0] [lt=72] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=12, avg_log_batch_cnt=0, total_group_log_size=1454, avg_group_log_size=121) [2025-01-08 00:52:01.071397] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:01.071451] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:52:01.072046] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=6661, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:01.072136] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1454) [2025-01-08 00:52:01.072157] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1319, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:01.072376] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:01.099807] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=34] get_max_applied_scn(scn={val:1736326320841974002, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2219647}, palf_committed_end_scn_:{val:1736326320841974003, v:0}, last_check_scn_:{val:1736326320841974003, v:0}, max_applied_cb_scn_:{val:1736326320841974002, v:0}}) [2025-01-08 00:52:01.099930] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=118] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2219647}, readable_log_tail:{lsn:2219647}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64885121, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:01.123371] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEF-0-0] [lt=26] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:52:01.123434] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CEF-0-0] [lt=52] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=66) [2025-01-08 00:52:01.134413] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=27] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:52:01.135488] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=37] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:01.150303] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=149][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.155119] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=52] gc stale ls task succ [2025-01-08 00:52:01.155288] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=39] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1003}, {id:1002}]) [2025-01-08 00:52:01.155580] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:01.155614] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-01-08 00:52:01.155668] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:01.155689] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:01.155702] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193825069330, flush_io_finished_ret_:0, flush_io_finished_round_:3807, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:01.156893] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:01.156942] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:01.158386] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87196-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:01.158466] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:01.158480] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:01.160888] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0D1-0-0] [lt=40][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:01.165611] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=31] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:01.165668] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:52:01.176260] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185453-0-0] [lt=20][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:01.176477] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=39] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:01.176501] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:52:01.176511] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:01.176524] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:01.176537] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193825087577, flush_io_finished_ret_:0, flush_io_finished_round_:1979, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:01.182442] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=13] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:52:01.182573] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=121] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:52:01.182589] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=13] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:52:01.182610] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=20] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:01.185825] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:52:01.185981] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=86] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:52:01.186399] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58561][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=15] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:52:01.191063] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=20] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:52:01.192804] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=77] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:52:01.192851] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=45] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:52:01.194210] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084882-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:52:01.194355] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084882-0-0] [lt=99] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:52:01.194429] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084882-0-0] [lt=70] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:52:01.198399] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=16] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:52:01.201666] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=48] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:52:01.203411] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=43] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:52:01.203463] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=49] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:52:01.204887] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084882-0-0] [lt=13] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:52:01.204993] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084882-0-0] [lt=52] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:52:01.205012] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084882-0-0] [lt=17] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:52:01.207921] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=13] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:52:01.213266] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=41] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:52:01.214721] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=41] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:52:01.214763] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=40] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:52:01.216099] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084882-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:52:01.216197] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084882-0-0] [lt=47] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:52:01.216301] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084882-0-0] [lt=101] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:52:01.216338] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=24] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:52:01.217106] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=763] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:01.217173] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=50] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:52:01.217186] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=12] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:01.217299] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084882-0-0] [lt=108] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:52:01.217320] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084882-0-0] [lt=19] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:52:01.217333] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=7] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:52:01.217340] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:01.217385] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084882-0-0] [lt=40] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:52:01.217404] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084882-0-0] [lt=18] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:52:01.217416] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=7] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:52:01.217423] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:01.217441] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084882-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:52:01.217454] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084882-0-0] [lt=12] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:52:01.219002] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=5] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:52:01.219046] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=40] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:52:01.219057] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=10] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:52:01.219089] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:52:01.219102] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:52:01.219110] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=7] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:52:01.219121] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=7] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:52:01.219135] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084882-0-0] [lt=13] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:52:01.224847] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=48, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:01.224887] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:01.227648] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5D94-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3539) [2025-01-08 00:52:01.227787] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=109] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8837, offset=2510827}, end_cursor:ObLogCursor{file_id=1, log_id=8838, offset=2511132}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d0c050, len:4117, log_data_len:305, seq:8837, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:01.227985] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:01.228134] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=149] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20936, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:01.228317] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E79-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:01.228354] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E79-0-0] [lt=19] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326320841974004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326320841974004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:01.253210] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E79-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1736326321243801000, start_id=1736326321243801000, end_id=1736326321243801001) [2025-01-08 00:52:01.254572] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBA7-0-0] [lt=60][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193826168403, cache_obj->added_lc()=false, cache_obj->get_object_id()=13699, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:01.254667] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=82] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:52:01.254701] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.261153] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8838, offset=2511132}, end_cursor:ObLogCursor{file_id=1, log_id=8839, offset=2511429}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d10050, len:3812, log_data_len:297, seq:8838, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:01.263158] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBA3-0-0] [lt=20][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:52:01.263501] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E79-0-0] [lt=21] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:01.263562] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E79-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326320841974004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326320841974004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:01.269844] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58674][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:01.270006] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:52:01.271912] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=64] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6724874240, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:01.271959] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:52:01.272674] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=37] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:01.279534] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:52:01.279667] WDIAG read_multi_row (ob_table_access_helper.h:238) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:52:01.280734] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [46273][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=75] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-01-08 00:52:01.281315] INFO delete_row (ob_table_access_helper.h:339) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=44] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326321279713) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326321279713) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:52:01.281396] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=26] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:52:01.281453] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:52:01.281504] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50445][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:52:01.283224] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=20] ====== check clog disk timer task ====== [2025-01-08 00:52:01.283252] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:52:01.292803] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58675][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:01.292869] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:52:01.294243] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8A9F-0-0] [lt=123] init point thread id with(&point=0x5572d50e7600, idx_=1331, point=[thread id=58675, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58675) [2025-01-08 00:52:01.300681] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] ====== check clog disk timer task ====== [2025-01-08 00:52:01.300719] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:52:01.310993] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1736326321310988, bytes: 0, bw: 0.000000 MB/s, add_ts: 1003601, add_bytes: 0 [2025-01-08 00:52:01.320236] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.320319] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.332015] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:01.339967] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE965-0-0] [lt=65] get number(ret=0, service_type_=0, range=1, base_id=1736326321339634000, start_id=1736326321339634000, end_id=1736326321339634001) [2025-01-08 00:52:01.345269] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326321345175], gts:1736326321344654000, latest_srr:[mts=1736326321345175]}) [2025-01-08 00:52:01.345345] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=74] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:01.345376] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326321345360], gts:1736326321344654000, latest_srr:[mts=1736326321345360]}) [2025-01-08 00:52:01.345728] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326321345714], gts:1736326321344654001, latest_srr:[mts=1736326321345714]}) [2025-01-08 00:52:01.354252] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:01.354848] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.378284] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=24] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:01.378590] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=51] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:01.386950] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:01.386987] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:52:01.387510] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:01.387727] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=136] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:10, bw:40361, [delay/us]:prepare:14, schedule:164, submit:508, rt:4071, total:4758) [2025-01-08 00:52:01.387747] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:01.387802] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:10, avg_bw:40362, [delay/us]: prepare:14, schedule:164, submit:508, device:4071, total:4758) [2025-01-08 00:52:01.387815] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:01.387856] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:80730, [delay/us]:prepare:20, schedule:558, submit:418, rt:4462, total:5459) [2025-01-08 00:52:01.387866] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:01.387901] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:80760, [delay/us]: prepare:20, schedule:558, submit:418, device:4462, total:5459) [2025-01-08 00:52:01.387912] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:01.387934] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:6144, iops:2, bw:12115, [delay/us]:prepare:8, schedule:21, submit:1, rt:2384, total:2415) [2025-01-08 00:52:01.387956] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:2, hold_mem:16384 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4286, iops:42, bw:181735, [delay/us]:prepare:10, schedule:233, submit:616, rt:3497, total:4358) [2025-01-08 00:52:01.387965] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=6, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49998, free_result_cnt=49998, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:01.388058] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=85] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:6144, avg_iops:2, avg_bw:12117, [delay/us]: prepare:8, schedule:21, submit:1, device:2384, total:2415) [2025-01-08 00:52:01.388157] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=88] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4287, avg_iops:43, avg_bw:181745, [delay/us]: prepare:10, schedule:233, submit:616, device:3497, total:4358) [2025-01-08 00:52:01.388169] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:01.388283] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=91] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4038, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:377, total:409) [2025-01-08 00:52:01.388316] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:01.388442] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=119] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4038, [delay/us]: prepare:0, schedule:0, submit:600000000, device:377, total:409) [2025-01-08 00:52:01.388464] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:01.388482] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:01.388492] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:01.388500] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:01.388508] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=258kB/s, net_out=251kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:01.390628] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=67] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4878, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:01.390684] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1564, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:01.390853] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=69, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:01.391903] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:01.394644] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:01.411380] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=138] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=13110) [2025-01-08 00:52:01.425228] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3540) [2025-01-08 00:52:01.425951] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=197] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:304K/70:285030 disk:304K/70:285030 t1m2:48K/11:157807 t1m2g0:48K/11:157807 t1002m2:184K/43:216803 t1002m2g0:184K/43:216803 t1001m2:72K/16:555855 t1001m2g0:72K/16:555855 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:01.440856] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:52:01.441246] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58629][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=25] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:52:01.454738] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=95] ====== tenant freeze timer task ====== [2025-01-08 00:52:01.455513] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.455471] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=7] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5325170693, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:52:01.455562] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:01.455574] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8bf5f1910) [2025-01-08 00:52:01.455582] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=7] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5325170693, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:52:01.456469] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:52:01.456556] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=81] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70704720, tenant_id=1001) [2025-01-08 00:52:01.456579] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:52:01.458589] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.459331] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326317289581, last_first_register_time_us=1736324449183670, register_time_us=1736326317289581, curr_time_us=1736326321459320) [2025-01-08 00:52:01.459440] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=67] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326321459401, last_submit_register_req_time_us=1736326321459401, register_parent_reason=1) [2025-01-08 00:52:01.459458] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:52:01.460543] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.463852] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4901-0-0] [lt=33][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18516, log_task={header:{begin_lsn:{lsn:2253668}, end_lsn:{lsn:2253790}, log_id:18516, min_scn:{val:1736326321344654000, v:0}, max_scn:{val:1736326321344654000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2253546}, prev_proposal_id:1, committed_end_lsn:{lsn:2253668}, data_checksum:1715074492, accum_checksum:3920254107, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326321360972, freeze_ts:1736326321360972, submit_ts:1736326321361042, flushed_ts:1736326321362631, gen_to_freeze cost time:0, gen_to_submit cost time:70, submit_to_flush cost time:1589}, fs_cb_begin_ts=1736326321463818, log_life_time=102846) [2025-01-08 00:52:01.473437] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:01.473664] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:52:01.481737] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:52:01.489685] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB622-0-0] [lt=147] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:01.489927] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB622-0-0] [lt=98] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326321339634001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326321339634001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:01.509400] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=65] rpc statistics(total_trans_msg_count=7, total_batch_msg_count=0) [2025-01-08 00:52:01.512850] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CC-0-0] [lt=32] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:01.512890] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CC-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326321344654001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326321344654001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:01.519258] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=108] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.542239] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=637] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.544755] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.555733] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:52:01.555779] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.598459] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18742109}], scn_array=[cnt:1, {val:1736326321344654003, v:0}], curr_size=122, accum_size=854, time_cost=11245) [2025-01-08 00:52:01.598799] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2220135}], scn_array=[cnt:1, {val:1736326321344654003, v:0}], curr_size=122, accum_size=854, time_cost=11468) [2025-01-08 00:52:01.599744] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBE17-0-0] [lt=99][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193826513576, cache_obj->added_lc()=false, cache_obj->get_object_id()=13700, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:01.655839] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.674418] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:01.674576] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:52:01.674912] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO [ratelimit] time: 1736326321674909, bytes: 885532506, bw: 0.263733 MB/s, add_ts: 1002339, add_bytes: 277191 [2025-01-08 00:52:01.702271] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2025-01-08 00:52:01.704605] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AB4-0-0] [lt=28] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:01.704783] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D54-0-0] [lt=25] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:52:01.711163] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2254203}, scn={val:1736326321344654004, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:52:01.711402] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=155] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324515701616000) [2025-01-08 00:52:01.711439] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=33] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2254203}, scn_:{val:1736326321344654004, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326321818995, first_handle_ts_:1736326321711383, replay_cost_:53, retry_cost_:-1, read_log_buf_:0x2ad96d004160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326321710915, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2254259}, next_to_submit_scn_:{val:1736326321344654005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2254147}, end_lsn:{lsn:2254259}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2254259}, readable_log_tail:{lsn:2254259}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64850509, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326321344654004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326321344654004, v:0}, data_checksum:2343689199, flag:59249}}, init_mode_version:0, accumulate_checksum:450731611, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326321344654004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18427, miss_cnt:0, cache_read_size:2254259, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=19, ret=0) [2025-01-08 00:52:01.755969] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.758552] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AB5-0-0] [lt=26] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:01.771703] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2245825, this={dio_aligned_buf:{buf_write_offset:1339, buf_padding_size:2757, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:24558, truncate_used_ts:5545}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2244608, buf_len=122, total_write_size=9732, total_write_size_after_dio=339968, ob_pwrite_used_ts=1462901, count=80, each_pwrite_cost=18286) [2025-01-08 00:52:01.782393] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D25DBAD49-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1736326321778016000, start_id=1736326321778016000, end_id=1736326321778016001) [2025-01-08 00:52:01.786271] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AB6-0-0] [lt=27] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:01.790281] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=37] failed to get result(ret=-4008) [2025-01-08 00:52:01.790486] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=23] check new jobs(ret=0, tenant_id=1002, is_oracle_tenant=false, job_infos=[{tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326232555425, start_date_:1736325329924481, last_date_:1736326229926861, this_date_:0, next_date_:1736327129924481, end_date_:64060560000000000, total_:6030537, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736410829924386, last_date_:0, this_date_:0, next_date_:1736410829924386, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1002, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324429937838, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:52:01.790614] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50473][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=126] check all tenants(ret=0) [2025-01-08 00:52:01.796998] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58674][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A31-0-0] [lt=45] init point thread id with(&point=0x5572d50e75c0, idx_=1330, point=[thread id=58674, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58674) [2025-01-08 00:52:01.811307] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AB9-0-0] [lt=8] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:01.824349] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584ABA-0-0] [lt=117] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:01.824420] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326321622836000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=201574, in_cluster_service=true, cluster_version={val:1736326321421833000, v:0}, min_cluster_version={val:1736326321421833000, v:0}, max_cluster_version={val:1736326321421833000, v:0}, get_cluster_version_err=0, cluster_version_delta=402577, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36904, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=576321397, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:52:01.831888] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=38] [APPLY STAT CB APPEND COST TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=156, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:52:01.831964] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=75] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=87846, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:52:01.832116] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=151] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:52:01.832125] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=-19004, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:52:01.832132] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=28, stat_interval=5000000, avg (count/cost)=68998, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:52:01.832495] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:01.836098] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584ABB-0-0] [lt=33] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:01.838559] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584ABC-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:01.840442] INFO pn_ratelimit (group.c:653) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=37] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:52:01.842935] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=16] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:52:01.842972] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:52:01.842978] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:52:01.842989] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=195) [2025-01-08 00:52:01.843007] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=21] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=193) [2025-01-08 00:52:01.846101] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=55] ts response task statistics(alloc_count=61078, free_count=30538) [2025-01-08 00:52:01.846152] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=40] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=557, get_gts_with_stc_cnt=22, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=21, try_wait_gts_elapse_cnt=0) [2025-01-08 00:52:01.846314] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=458, get_gts_with_stc_cnt=60, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=61, try_wait_gts_elapse_cnt=0) [2025-01-08 00:52:01.853135] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=36] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:01.853832] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=88] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=651907072, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=421834752, wash_ratio=6, tenant_id=1001) [2025-01-08 00:52:01.853911] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=44] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:52:01.853932] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2894, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:52:01.853991] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:52:01.853990] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:52:01.854043] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:52:01.854056] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:52:01.854089] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:01.854138] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=43] temporary file module use no blocks [2025-01-08 00:52:01.854232] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=92] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:01.854249] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:01.854269] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193825767796, flush_io_finished_ret_:0, flush_io_finished_round_:1984, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:01.854545] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:01.855831] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15282328479, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:01.856133] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=119] recycle one thread(this=0x5572d527f900, total=5, remain=4) [2025-01-08 00:52:01.856162] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=28] inc thread count(*this={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15282328479, tenant_id:500}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:01.856176] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15214495972, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:52:01.856309] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] recycle one thread(this=0x2ad8881fc3b0, total=6, remain=5) [2025-01-08 00:52:01.856322] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15214495972, tenant_id:1}, cur_thread_count=6, cnt=-1, new_thread_count=5) [2025-01-08 00:52:01.856388] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=65] try inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:8011319828, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:01.856452] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] recycle one thread(this=0x2ad8c0eb0e70, total=5, remain=4) [2025-01-08 00:52:01.856464] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] inc thread count(*this={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:8011319828, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:01.856480] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.865123] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDF-0-0] [lt=14] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:01.865171] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CDF-0-0] [lt=44] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:01.868142] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:01.872651] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2254381}], scn_array=[cnt:1, {val:1736326321846247001, v:0}], curr_size=122, accum_size=1201, time_cost=10017) [2025-01-08 00:52:01.875312] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:01.875689] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=23] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1557072, replace_num=15728) [2025-01-08 00:52:01.875716] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:52:01.876912] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=58] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1447022, clean_num=31457) [2025-01-08 00:52:01.878607] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=159] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:01.879107] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:01.890364] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=24] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:01.890390] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:52:01.890442] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84995-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:01.890513] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84995-0-0] [lt=15] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:01.892955] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:01.893851] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584ABD-0-0] [lt=26] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:01.894302] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC43A-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:01.894854] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=388] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:01.896803] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.896906] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.896941] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.897324] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:01.906414] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8B-00062B2D198BB258-0-0] [lt=733] get number(ret=0, service_type_=0, range=1, base_id=1736326321905789000, start_id=1736326321905789000, end_id=1736326321905789001) [2025-01-08 00:52:01.911405] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584ABE-0-0] [lt=40] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:01.946738] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC7-0-0] [lt=42][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:01.951263] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC7-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:01.956601] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:01.963812] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A490B-0-0] [lt=62] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:01.963899] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A490B-0-0] [lt=73] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1323, avg_group_log_size=120) [2025-01-08 00:52:01.963921] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A490B-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1323) [2025-01-08 00:52:01.965770] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=218] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3274, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:01.965796] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1323) [2025-01-08 00:52:01.965818] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=200, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:01.965884] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=93, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:01.981592] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BAB-0-0] [lt=47] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:01.981691] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D7-0-0] [lt=43] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326322002675, keepalive_ts:-1}) [2025-01-08 00:52:01.981718] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D7-0-0] [lt=25] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326322002675, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:01.982373] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF36-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326322001967, keepalive_ts:-1}) [2025-01-08 00:52:01.982404] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF36-0-0] [lt=30] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326322001967, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:01.985515] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16E9CA78-0-0] [lt=94] get number(ret=0, service_type_=0, range=1, base_id=1736326321984466000, start_id=1736326321984466000, end_id=1736326321984466001) [2025-01-08 00:52:01.985698] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16E9CA78-0-0] [lt=42] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3963, avg_group_log_size=396) [2025-01-08 00:52:01.986510] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FF-0-0] [lt=41] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326322009359, keepalive_ts:-1}) [2025-01-08 00:52:01.986535] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0FF-0-0] [lt=25] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326322009359, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:01.987084] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF72-0-0] [lt=15] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326322094120, keepalive_ts:-1}) [2025-01-08 00:52:01.987102] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF72-0-0] [lt=17] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326322094120, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:01.987134] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D6-0-0] [lt=11] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326322002519, keepalive_ts:-1}) [2025-01-08 00:52:01.987145] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D6-0-0] [lt=11] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326322002519, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:01.988884] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=78, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:01.988949] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:01.989067] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:01.989099] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:52:01.989165] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=62] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1098) [2025-01-08 00:52:01.989241] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=72] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:01.990257] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3963) [2025-01-08 00:52:01.990838] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB2A-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:01.990896] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB2A-0-0] [lt=57] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5737, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:01.990905] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB2A-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=18, avg_log_submit_to_flush_time=2486, avg_log_submit_to_slide_time=5718) [2025-01-08 00:52:01.990968] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=100] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:01.991029] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18330, miss_cnt:0, cache_read_size:2242095, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:01.991090] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=67] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=145, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:01.991105] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=15] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:01.991508] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=51] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2728, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:01.991536] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:01.991568] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A33-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=88, avg_log_submit_to_flush_time=4021, avg_log_submit_to_slide_time=5722) [2025-01-08 00:52:01.991587] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=762, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:01.991636] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3963, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:01.991697] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:01.991720] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:01.992023] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3807, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:01.992104] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1643) [2025-01-08 00:52:01.992120] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=534, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:01.992153] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1643) [2025-01-08 00:52:01.992165] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:01.995027] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685523-0-0] [lt=17] MediumCheckTask(cost_ts=9) [2025-01-08 00:52:01.995777] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A96EA-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:01.995810] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A96EA-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=5865, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:01.995873] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:01.996398] INFO [LIB] stat (utility.h:1210) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A96E9-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:01.996418] INFO [LIB] stat (utility.h:1210) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A96E9-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=6722, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:01.996427] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58646][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A96E9-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=107, avg_log_submit_to_flush_time=4660, avg_log_submit_to_slide_time=6614) [2025-01-08 00:52:01.996833] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1521, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:02.019669] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=67, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:02.019779] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=97] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9862, avg_group_log_size=657) [2025-01-08 00:52:02.024801] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3C10-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11137) [2025-01-08 00:52:02.027371] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=3355, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:02.027415] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9862) [2025-01-08 00:52:02.027439] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=701, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:02.027484] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=25, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:02.027499] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=5692, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:02.027507] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=6] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=102, avg_log_submit_to_flush_time=4302, avg_log_submit_to_slide_time=5589) [2025-01-08 00:52:02.027528] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=26, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:02.027674] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=196] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9273, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:02.028777] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326321339634004, v:0}, new_max_version={val:1736326321339634004, v:0}, last_min_version={val:1736326321339634002, v:0}, last_max_version={val:1736326321339634002, v:0}, total_time_us=11521, delta_us=689141) [2025-01-08 00:52:02.028855] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326321339634004, v:0}, min_version={val:1736326321339634004, v:0}, max_version={val:1736326321339634004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:02.056729] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.059575] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BC3-0-0] [lt=33] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:02.068760] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB2B-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=19, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:02.068900] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB2B-0-0] [lt=138] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=24430, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:02.068902] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1445, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:02.068916] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58641][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB2B-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=25, avg_log_submit_to_flush_time=3375, avg_log_submit_to_slide_time=24405) [2025-01-08 00:52:02.068939] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18431, miss_cnt:0, cache_read_size:2254747, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:02.072415] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BAD-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:02.072476] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BAD-0-0] [lt=52] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:02.074804] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2133, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:02.074837] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:02.074861] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=251, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:02.074914] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=62, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:02.075906] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:52:02.077366] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:02.091980] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=30] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:52:02.092041] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=48] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=97) [2025-01-08 00:52:02.092589] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=38] [APPLY STAT CB APPEND COST TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=82, this=0x2ad934926280, extra_info="") [2025-01-08 00:52:02.092650] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=60] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=43870, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:52:02.092664] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=13] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:52:02.092675] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB EXECUTE TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=-8020, this=0x2ad934926910, extra_info="") [2025-01-08 00:52:02.092685] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB TOTAL TIME](cur_stat_count=21, stat_interval=5000000, avg (count/cost)=35933, this=0x2ad934926b40, extra_info="") [2025-01-08 00:52:02.117355] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:2254747}, scn={val:1736326321846247004, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326322068736, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2254747}, next_to_submit_scn_:{val:1736326321846247004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2254625}, end_lsn:{lsn:2254747}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2254747}, readable_log_tail:{lsn:2254747}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64850021, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326321846247003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326321846247003, v:0}, data_checksum:3348837161, flag:7449}}, init_mode_version:0, accumulate_checksum:924292035, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326321846247003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18431, miss_cnt:0, cache_read_size:2254747, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:02.135673] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=47] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:02.155361] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:02.155382] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:52:02.155390] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:02.155867] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}, {id:1003}, {id:1002}, {id:1001}]) [2025-01-08 00:52:02.156008] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:02.156035] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-01-08 00:52:02.156044] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:02.156058] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:02.156071] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193826069574, flush_io_finished_ret_:0, flush_io_finished_round_:3808, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:02.156947] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=100][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.158351] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0D4-0-0] [lt=47][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:02.160738] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8719C-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:02.160858] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=90][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:02.160900] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:02.165917] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:02.165974] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:52:02.176942] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=39] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:52:02.178981] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185454-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:02.179398] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:02.179425] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-01-08 00:52:02.179435] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:02.179448] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:02.179460] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193826090408, flush_io_finished_ret_:0, flush_io_finished_round_:1980, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:02.198638] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=32] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:52:02.219777] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AF7A-0-0] [lt=115] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.793968391418457031e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=389, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.608244076411260721e+02) [2025-01-08 00:52:02.227843] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=75, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:02.227900] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:02.229030] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=79] get_max_applied_scn(scn={val:1736326322022412000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18743687}, palf_committed_end_scn_:{val:1736326322022412001, v:0}, last_check_scn_:{val:1736326322022412001, v:0}, max_applied_cb_scn_:{val:1736326322022412000, v:0}}) [2025-01-08 00:52:02.229127] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=90] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18743687}, readable_log_tail:{lsn:18743687}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48361081, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:02.229329] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=175] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:52:02.229433] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3962) [2025-01-08 00:52:02.231402] INFO [LIB] stat (utility.h:1210) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A36-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:02.231439] INFO [LIB] stat (utility.h:1210) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A36-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5618, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:02.231711] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:02.255288] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=15] LogLoopThread round_cost_time(us)(round_cost_time=67) [2025-01-08 00:52:02.257136] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.265097] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:52:02.265707] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58671][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=46] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:52:02.272926] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=38] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:02.276297] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=48] ====== check clog disk timer task ====== [2025-01-08 00:52:02.276334] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=29] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:52:02.277123] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=59] ====== traversal_flush timer task ====== [2025-01-08 00:52:02.277289] INFO [MDS] flush (mds_table_mgr.cpp:195) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B6-0-0] [lt=78] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326322022412001, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326317803522001, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326330087570000, latest_log_ts:{val:1736326313644270001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:02.277417] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B6-0-0] [lt=95] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326317629447000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326317803522001, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:02.277499] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B6-0-0] [lt=55] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326317803522002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326317803522001, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:02.277578] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B6-0-0] [lt=52] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326317454295003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326317803522001, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:02.277702] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B6-0-0] [lt=119] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:52:02.279434] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58672][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3C13-0-0] [lt=40] init point thread id with(&point=0x5572d50e7540, idx_=1328, point=[thread id=58672, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58672) [2025-01-08 00:52:02.280272] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:52:02.281108] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBA9-0-0] [lt=43][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193827194941, cache_obj->added_lc()=false, cache_obj->get_object_id()=13701, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:02.282002] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:02.297268] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=110] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8839, offset=2511429}, end_cursor:ObLogCursor{file_id=1, log_id=8840, offset=2511734}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d14050, len:3515, log_data_len:305, seq:8839, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:02.297714] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:52:02.297745] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7A-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:02.297815] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7A-0-0] [lt=51] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326322022412001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326322022412001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:02.298021] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58585][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=145] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:52:02.311133] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=41] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8840, offset=2511734}, end_cursor:ObLogCursor{file_id=1, log_id=8841, offset=2512031}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d18050, len:3210, log_data_len:297, seq:8840, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:02.311544] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7A-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:02.311587] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7A-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326322022412001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326322022412001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:02.318664] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1736326322318660, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007672, add_bytes: 0 [2025-01-08 00:52:02.319466] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.319521] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.322268] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=39] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8841, offset=2512031}, end_cursor:ObLogCursor{file_id=1, log_id=8842, offset=2512328}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d1c050, len:2913, log_data_len:297, seq:8841, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:02.324061] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8842, offset=2512328}, end_cursor:ObLogCursor{file_id=1, log_id=8843, offset=2512625}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d20050, len:2616, log_data_len:297, seq:8842, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:02.332761] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=80] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:02.341589] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C79C2B6-0-0] [lt=90] get number(ret=0, service_type_=0, range=1, base_id=1736326322338447000, start_id=1736326322338447000, end_id=1736326322338447001) [2025-01-08 00:52:02.347104] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=80] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:02.354907] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=16637952) [2025-01-08 00:52:02.357327] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=135][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.378822] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:02.379677] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=43] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:02.384528] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BB3-0-0] [lt=17][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18423, log_task={header:{begin_lsn:{lsn:2242339}, end_lsn:{lsn:2242461}, log_id:18423, min_scn:{val:1736326321839865004, v:0}, max_scn:{val:1736326321839865004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2242217}, prev_proposal_id:1, committed_end_lsn:{lsn:2242339}, data_checksum:940009244, accum_checksum:1286027010, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326322283651, freeze_ts:1736326322283651, submit_ts:1736326322283664, flushed_ts:1736326322286638, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:2974}, fs_cb_begin_ts=1736326322384500, log_life_time=100849) [2025-01-08 00:52:02.390959] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1944, avg_group_log_size=216) [2025-01-08 00:52:02.391004] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1944) [2025-01-08 00:52:02.391854] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:02.391920] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44865, [delay/us]:prepare:9, schedule:86, submit:485, rt:2384, total:2966) [2025-01-08 00:52:02.391937] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:02.391999] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44867, [delay/us]: prepare:9, schedule:86, submit:485, device:2384, total:2966) [2025-01-08 00:52:02.392052] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:02.392096] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4915, iops:15, bw:73416, [delay/us]:prepare:15, schedule:159, submit:150, rt:2457, total:2782) [2025-01-08 00:52:02.392107] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:02.392303] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=182] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4915, avg_iops:15, avg_bw:73404, [delay/us]: prepare:15, schedule:159, submit:150, device:2457, total:2782) [2025-01-08 00:52:02.392370] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:02.392395] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16311, [delay/us]:prepare:7, schedule:49, submit:1, rt:1884, total:1941) [2025-01-08 00:52:02.392418] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4306, iops:39, bw:167190, [delay/us]:prepare:7, schedule:218, submit:211, rt:2348, total:2785) [2025-01-08 00:52:02.392437] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:02.392838] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=391] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16306, [delay/us]: prepare:7, schedule:49, submit:1, device:1884, total:1941) [2025-01-08 00:52:02.392860] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4306, avg_iops:39, avg_bw:167149, [delay/us]: prepare:7, schedule:218, submit:211, device:2348, total:2785) [2025-01-08 00:52:02.392872] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:02.392895] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4077, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:358, total:389) [2025-01-08 00:52:02.392921] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:02.392997] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4077, [delay/us]: prepare:0, schedule:0, submit:600000000, device:358, total:389) [2025-01-08 00:52:02.393019] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:02.393042] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:02.393054] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:02.393061] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:02.393086] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SERVER] net_in=313kB/s, net_out=229kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:02.393519] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:02.395464] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=170] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:02.395714] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3115, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:02.395909] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=125] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=852, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:02.395909] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=131] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=60, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:02.419904] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9819) [2025-01-08 00:52:02.427890] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] get number(ret=0, service_type_=0, range=1, base_id=1736326322426966000, start_id=1736326322426966000, end_id=1736326322426966001) [2025-01-08 00:52:02.428095] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3962) [2025-01-08 00:52:02.428474] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=41] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:296K/68:161433 disk:296K/68:161433 t1m2:44K/10:53387 t1m2g0:44K/10:53387 t1002m2:180K/43:193624 t1002m2g0:180K/43:193624 t1001m2:72K/15:141182 t1001m2g0:72K/15:141182 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:02.438769] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=41] ====== traversal_flush timer task ====== [2025-01-08 00:52:02.438928] INFO [MDS] flush (mds_table_mgr.cpp:195) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F4C-0-0] [lt=39] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326322338447000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326329284252000, latest_log_ts:{val:1736326312645069001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:02.439036] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50561][T1001_TimerWK3][T1001][YB42C0A89F89-00062B2D1D384F4C-0-0] [lt=102] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:52:02.447929] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:52:02.450962] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58669][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=53] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:52:02.453449] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=53] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:52:02.455407] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBAB-0-0] [lt=46][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193827369195, cache_obj->added_lc()=false, cache_obj->get_object_id()=13702, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:02.456944] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5328131153, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:52:02.457075] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=50] recycle one thread(this=0x2ad8885abcf0, total=4, remain=3) [2025-01-08 00:52:02.457157] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=80] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5328131153, tenant_id:500}, cur_thread_count=4, cnt=-1, new_thread_count=3) [2025-01-08 00:52:02.457505] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.458486] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.459962] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.462469] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326318372607, last_first_register_time_us=1736324454612308, register_time_us=1736326318372607, curr_time_us=1736326322462466) [2025-01-08 00:52:02.462539] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326322462502, last_submit_register_req_time_us=1736326322462502, register_parent_reason=1) [2025-01-08 00:52:02.462555] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:52:02.464887] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBB0-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326322463924000, start_id=1736326322463924000, end_id=1736326322463924001) [2025-01-08 00:52:02.470753] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=6] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:52:02.480582] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:52:02.482582] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=93] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:02.500123] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=48] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:52:02.500163] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=35] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=139, cache_node_num=139) [2025-01-08 00:52:02.500237] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=71] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=139, cache_node_num=139) [2025-01-08 00:52:02.500249] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:02.503285] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=139, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}, {obj_id:2083, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221102}, {obj_id:2084, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2085, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2086, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2087, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2088, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2089, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:52:02.503832] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=543] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:02.504288] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58677][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:02.505256] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:52:02.519268] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.542288] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.543981] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=179] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.544671] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB623-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:02.544724] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB623-0-0] [lt=32] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326322345307001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326322345307001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:02.557664] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=99][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.558636] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CD-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:02.558681] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CD-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326322353812001, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326322353812001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:02.566235] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=27] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:52:02.595922] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58677][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8AC5-0-0] [lt=28] init point thread id with(&point=0x5572d50e7680, idx_=1333, point=[thread id=58677, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58677) [2025-01-08 00:52:02.619952] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AF80-0-0] [lt=99] need check schema(new_schema_version=1736326191428712, cached_tenant_schema_version=1736326191428712, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=update __all_weak_read_service set min_version=?, max_version=? where tenant_id = ? and level_id = ? and level_value = ? and min_version = ? and max_version = ?) [2025-01-08 00:52:02.620790] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=144] get number(ret=0, service_type_=0, range=1, base_id=1736326322616977000, start_id=1736326322616977000, end_id=1736326322616977001) [2025-01-08 00:52:02.627805] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] submit log success(service_type_=0, last_id=1736326339254996000, limited_id=1736326339254996000) [2025-01-08 00:52:02.627830] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] submit log with lock success(service_type_=0, last_id=1736326339254996000, limited_id=1736326339254996000) [2025-01-08 00:52:02.630724] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=20] update id meta(ret=0, service_type=0, limited_id=1736326339254996000, latest_scn={val:1736326322625057001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339254996000, latest_log_ts:{val:1736326322625057001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:52:02.631010] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=284] handle submit callback(service_type_=0, success=true, used_ts=3206, log_ts={val:1736326322625057001, v:0}, limited_id=1736326339254996000) [2025-01-08 00:52:02.631032] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=21] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:52:02.631043] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=11] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:52:02.632121] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326322631942000, start_id=1736326322631942000, end_id=1736326322631942001) [2025-01-08 00:52:02.657809] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.669419] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=141] submit log success(service_type_=0, last_id=1736326339284252000, limited_id=1736326339284252000) [2025-01-08 00:52:02.669454] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] submit log with lock success(service_type_=0, last_id=1736326339284252000, limited_id=1736326339284252000) [2025-01-08 00:52:02.671278] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BC4-0-0] [lt=29][errcode=-4006] not init(ret=-4006) [2025-01-08 00:52:02.671332] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:480) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BC4-0-0] [lt=46] finish scan map holding items(scan_cnt=0) [2025-01-08 00:52:02.671496] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BC5-0-0] [lt=12] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:02.673441] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=246] update id meta(ret=0, service_type=0, limited_id=1736326339284252000, latest_scn={val:1736326322666668001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339284252000, latest_log_ts:{val:1736326322666668001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:52:02.673564] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=120] handle submit callback(service_type_=0, success=true, used_ts=4155, log_ts={val:1736326322666668001, v:0}, limited_id=1736326339284252000) [2025-01-08 00:52:02.673578] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=13] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:52:02.673587] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:52:02.675444] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=10] get number(ret=0, service_type_=0, range=1, base_id=1736326322674956000, start_id=1736326322674956000, end_id=1736326322674956001) [2025-01-08 00:52:02.675735] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=123] PNIO [ratelimit] time: 1736326322675732, bytes: 885812767, bw: 0.267058 MB/s, add_ts: 1000823, add_bytes: 280261 [2025-01-08 00:52:02.680791] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:52:02.683209] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:02.702547] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=35] current monitor number(seq_=-1) [2025-01-08 00:52:02.758027] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.760422] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=26] ====== tenant freeze timer task ====== [2025-01-08 00:52:02.771625] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5990) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=28] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000) [2025-01-08 00:52:02.771679] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=53] get multi version start(ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.771757] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.771816] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.771909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.772020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.772123] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.772253] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.772354] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.772599] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=115] get multi version start(ls_id={id:1}, tablet_id={id:60259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.772701] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.772798] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.772861] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.772960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773023] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773031] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=49] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:52:02.773083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773150] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:101109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773316] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=104] get multi version start(ls_id={id:1}, tablet_id={id:284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773418] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:50240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773778] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101098}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773896] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.773956] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774104] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:60305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774240] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774314] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101036}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774494] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774553] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101064}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774731] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.774888] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.775110] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:101080}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.775184] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.775319] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.775387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.775524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.775648] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.775713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.775859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776146] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776320] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101083}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776386] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101062}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776507] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776635] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776721] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.776908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777129] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101018}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777322] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777459] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777585] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101034}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101084}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.777958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778152] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778331] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778459] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778663] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101068}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778733] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.778971] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779037] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779231] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779363] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779844] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.779971] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780158] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=73] get multi version start(ls_id={id:1}, tablet_id={id:101073}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=86] get multi version start(ls_id={id:1}, tablet_id={id:60383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780362] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780421] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780539] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101099}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101092}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780774] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780833] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.780891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.781005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.781089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.781166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.782112] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.782261] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.782328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101031}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.782389] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.782448] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.782557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.782622] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.782681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=328] get multi version start(ls_id={id:1}, tablet_id={id:517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783242] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783311] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:101103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783506] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783565] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783675] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783745] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783805] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783923] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.783982] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784099] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784158] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101067}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784407] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784473] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784536] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784658] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101027}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784718] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784778] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784897] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.784994] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785314] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:50003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785376] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785641] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785701] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785819] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101082}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785937] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.785995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786172] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786290] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786355] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786416] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786897] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.786956] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787250] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787334] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101057}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787413] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787473] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787590] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787865] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.787925] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788101] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=118] get multi version start(ls_id={id:1}, tablet_id={id:50391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788344] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788408] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101033}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788690] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788753] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788813] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.788990] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.789050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101066}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.789433] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=320] get multi version start(ls_id={id:1}, tablet_id={id:50305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.789620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=132] get multi version start(ls_id={id:1}, tablet_id={id:50121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.789682] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.789741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.789801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790071] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=182] get multi version start(ls_id={id:1}, tablet_id={id:60119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790170] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790313] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790440] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790559] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101030}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101094}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101020}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790795] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101044}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:100002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.790946] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:60452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.791008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.791067] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.791166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:50167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.793656] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.793746] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.794589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.794716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=65] get multi version start(ls_id={id:1}, tablet_id={id:50515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.794889] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=114] get multi version start(ls_id={id:1}, tablet_id={id:290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.794954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.795015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.795305] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=209] get multi version start(ls_id={id:1}, tablet_id={id:371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.795373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.795433] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.795716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=30] get multi version start(ls_id={id:1}, tablet_id={id:50455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.795786] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101013}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.795847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.795907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.796064] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.796132] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.796340] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.796410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.796500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:60398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.796682] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=125] get multi version start(ls_id={id:1}, tablet_id={id:101072}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.796743] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.796803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.796939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101088}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797223] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101069}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797325] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797407] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797721] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.797933] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798150] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=59] get multi version start(ls_id={id:1}, tablet_id={id:191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798287] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798353] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101060}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798462] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798529] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101100}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798663] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.798965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.799024] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101022}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.799084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.799144] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.799262] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.799330] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.799625] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=142] get multi version start(ls_id={id:1}, tablet_id={id:482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.799692] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.799752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.799887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800086] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800150] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800328] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800519] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800821] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800880] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.800999] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101035}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:5}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801462] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801585] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801646] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101058}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801840] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801921] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101081}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.801995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802055] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101086}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101052}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802312] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101019}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802434] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802513] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802575] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101079}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802634] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802752] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.802909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.803021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.803084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.803143] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.803259] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.803321] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.803380] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.803439] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.803498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.805598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.805865] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.805936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.806022] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:50430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.806083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.806212] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.806281] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.806341] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.806400] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.806484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.806863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.807041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.807145] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:60393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.807294] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.807393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:50366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.807538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.807633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.807881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=189] get multi version start(ls_id={id:1}, tablet_id={id:60148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.807976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.808066] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.808157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.808334] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=119] get multi version start(ls_id={id:1}, tablet_id={id:426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.808427] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.808637] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.808732] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.809320] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=266] get multi version start(ls_id={id:1}, tablet_id={id:235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.809427] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:50336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.809518] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.809607] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.809876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=92] get multi version start(ls_id={id:1}, tablet_id={id:110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.809993] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=54] get multi version start(ls_id={id:1}, tablet_id={id:60254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.810163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.810334] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=108] get multi version start(ls_id={id:1}, tablet_id={id:60146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.810455] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.810573] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=63] get multi version start(ls_id={id:1}, tablet_id={id:50423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.810667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.810761] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.810852] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.810943] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.811034] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.811126] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.811278] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.811405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=61] get multi version start(ls_id={id:1}, tablet_id={id:50452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.811509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:60469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.811602] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.811693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.811842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.811939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.812030] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.812120] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.812343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.812442] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.812543] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.812634] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.812726] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.812817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.813002] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.813100] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.813266] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.813367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:50189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.813514] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.813617] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:50492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.813708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.814588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.814759] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=110] get multi version start(ls_id={id:1}, tablet_id={id:60332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.814884] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=59] get multi version start(ls_id={id:1}, tablet_id={id:60405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.815005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=246] get multi version start(ls_id={id:1}, tablet_id={id:60213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816299] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:60375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816460] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816526] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816584] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101043}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816820] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.816963] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817053] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817315] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=83] get multi version start(ls_id={id:1}, tablet_id={id:149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817434] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817493] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817552] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817612] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817670] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817729] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:60246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.817962] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101032}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101089}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818139] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818281] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818453] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818572] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818753] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818819] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818880] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.818940] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819001] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819062] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819123] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819236] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819305] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819366] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819427] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819599] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819790] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819912] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.819972] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820032] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820093] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101028}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820270] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820337] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820486] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820546] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820607] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820667] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101021}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.820968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821434] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821495] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821798] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821919] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.821979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822051] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822280] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822357] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822419] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101037}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822479] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822721] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822782] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101054}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822902] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.822962] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823052] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823116] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823226] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823296] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823418] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823479] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101041}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823599] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823658] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823719] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101097}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823778] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.823889] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101024}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824316] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824546] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824607] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824756] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:50517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824819] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.824994] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101055}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.825052] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.825110] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.825168] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.825282] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.825345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.825469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.825557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.825620] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.826498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.826598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:50433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.826688] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.826777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.826867] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.826956] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827052] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:101063}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827362] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=144] get multi version start(ls_id={id:1}, tablet_id={id:60397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827472] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=68] get multi version start(ls_id={id:1}, tablet_id={id:3}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827675] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:60360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827735] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827793] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827851] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.827965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828046] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101040}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828467] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828529] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828683] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828743] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101047}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828860] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.828958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829146] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829307] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829369] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829486] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829543] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829601] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101059}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829658] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829716] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829774] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829911] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.829976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830034] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830093] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830280] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830384] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830513] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830571] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830629] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830685] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.830988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831044] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831101] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831289] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831352] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831465] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831537] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831671] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831784] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831840] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831896] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.831953] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101046}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832121] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832177] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832378] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832574] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832743] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.832984] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833048] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833110] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833171] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833259] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=107] get multi version start(ls_id={id:1}, tablet_id={id:50291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833492] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101045}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833495] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:02.833552] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833671] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833874] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.833995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834083] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:50243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834153] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834341] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834406] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834465] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101078}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834642] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834718] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834861] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101049}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.834955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835078] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835255] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835322] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101101}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835382] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835500] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835575] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835707] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835767] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.835878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836049] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=114] get multi version start(ls_id={id:1}, tablet_id={id:213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836110] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836170] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836411] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101091}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836829] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101039}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836888] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.836946] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101029}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=29] get multi version start(ls_id={id:1}, tablet_id={id:283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837281] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837349] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837469] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837821] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837880] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.837958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.838036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839344] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839404] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839640] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839699] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.839828] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840066] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840132] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840290] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101095}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840359] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101070}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840419] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840479] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840538] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101075}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.840849] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841051] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=56] get multi version start(ls_id={id:1}, tablet_id={id:60373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841130] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:60197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841248] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841316] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101077}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841430] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101071}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841600] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101093}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841713] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841826] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.841997] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842085] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101065}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842420] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842477] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842640] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842830] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.842902] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843072] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843136] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843344] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843572] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464A-0-0] [lt=42] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2025-01-08 00:52:02.843623] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843680] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843812] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101014}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843933] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.843990] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844047] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844102] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844159] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844342] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844417] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844542] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844623] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844745] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844809] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844869] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.844988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845048] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845290] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=68] get multi version start(ls_id={id:1}, tablet_id={id:520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845352] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845323] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464A-0-0] [lt=28] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2025-01-08 00:52:02.845404] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1010) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464A-0-0] [lt=77] no transfer partition job [2025-01-08 00:52:02.845412] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845468] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1040) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464A-0-0] [lt=8] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer=true) [2025-01-08 00:52:02.845472] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845617] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845737] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845796] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845915] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.845974] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846033] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846092] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846264] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846330] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846389] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101050}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846448] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846833] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.846950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847068] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847234] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101056}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847302] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:60511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847627] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847686] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847745] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847863] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.847923] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848134] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:284) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464A-0-0] [lt=16] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-01-08 00:52:02.848171] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:493) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464A-0-0] [lt=24] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2025-01-08 00:52:02.848263] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:157) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464A-0-0] [lt=83] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=3, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1003}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone3", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1736324440080711, last_statistic_bg_stat_time=1736326192603087, last_statistic_schema_version=1736326192037600, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-01-08 00:52:02.847982] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101023}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848431] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=51] handle gts request statistics(total_rt=2873, total_cnt=279, avg_rt=1.026071428571428612e+01, avg_cnt=2.789999999999999858e+01) [2025-01-08 00:52:02.848493] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=451] get multi version start(ls_id={id:1}, tablet_id={id:50317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101048}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848733] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101026}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848850] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848909] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.848968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.849027] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.849113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.850024] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE970-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1736326322849540000, start_id=1736326322849540000, end_id=1736326322849540001) [2025-01-08 00:52:02.850426] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.850509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:50354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.850574] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.850686] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.850755] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.850930] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.851008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.851069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.851128] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.851291] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.851559] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.851919] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.851984] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852042] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101025}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852102] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852161] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852291] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852360] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852420] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852501] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852577] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:4}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852637] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:100005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852718] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853CE-0-0] [lt=90] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:52:02.852749] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853CE-0-0] [lt=24] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:52:02.852755] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852814] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.852989] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.853120] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.853428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.853495] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.853554] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.853687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.853907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.853972] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.854111] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.854247] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:52:02.854333] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.854408] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.854419] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:52:02.854491] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.854500] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] get_min_unreplayed_log_info(lsn={lsn:2242949}, scn={val:1736326322345307004, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326322701487, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2242949}, next_to_submit_scn_:{val:1736326322345307004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2242827}, end_lsn:{lsn:2242949}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2243071}, readable_log_tail:{lsn:2243071}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64861697, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326322345307003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326322345307003, v:0}, data_checksum:2124447694, flag:47486}}, init_mode_version:0, accumulate_checksum:2383140506, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326322345307003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18337, miss_cnt:0, cache_read_size:2242949, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:02.854760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.854838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:101085}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.854914] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855045] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855111] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855333] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855352] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=364] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:02.855402] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855524] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855606] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855746] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:60443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.855968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=30] get multi version start(ls_id={id:1}, tablet_id={id:132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.856030] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101087}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.856090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.856382] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:02.856493] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=343] get multi version start(ls_id={id:1}, tablet_id={id:60403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.856556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.856661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.856732] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.856792] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.856959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=95] get multi version start(ls_id={id:1}, tablet_id={id:50427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857091] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857153] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857294] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857367] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15219506351, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:02.857394] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:02.857429] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857405] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8881fc3b0, thread=0x2ad8b50e9ed0) [2025-01-08 00:52:02.857465] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=58] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15219506351, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:02.857490] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101051}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857550] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:100006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857610] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857670] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857830] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.857951] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.858011] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.858071] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.858131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101061}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.858159] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.858309] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.858393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.858783] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=326] get multi version start(ls_id={id:1}, tablet_id={id:60211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.858847] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.858908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.858969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101076}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.859028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.859087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.859146] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.859291] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.859429] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.859496] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.859555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.859614] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.859671] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.870131] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:02.871353] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=28] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:52:02.872847] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE0-0-0] [lt=314] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:02.872875] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE0-0-0] [lt=25] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:02.859729] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873124] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13276] get multi version start(ls_id={id:1}, tablet_id={id:60446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873247] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873315] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873433] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873490] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873605] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873662] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101074}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.873948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.874006] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.874626] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:52:02.874941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875021] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:101053}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875139] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875252] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875381] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875442] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=107] get multi version start(ls_id={id:1}, tablet_id={id:60142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875897] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.875955] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876012] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60322}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876415] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876534] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876591] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876649] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876763] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50322}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876821] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876878] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876936] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.876993] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877128] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877268] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877335] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877394] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877625] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877682] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877740] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877911] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.877969] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.878307] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=281] get multi version start(ls_id={id:1}, tablet_id={id:158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.878388] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.878449] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.878742] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=237] get multi version start(ls_id={id:1}, tablet_id={id:60508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.878801] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.878859] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.878916] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:322}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.878973] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879030] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101042}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879136] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879407] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879561] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879624] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879738] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879796] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879853] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879910] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.879968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880025] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880082] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101096}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880077] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC43D-0-0] [lt=56][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:02.880140] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880197] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880362] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=110] get multi version start(ls_id={id:1}, tablet_id={id:60411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880423] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880524] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:02.880537] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880582] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:02.880595] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880652] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880778] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880841] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.880898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.881039] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:52:02.881023] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.881272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=116] get multi version start(ls_id={id:1}, tablet_id={id:50197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.881338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.881512] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=102] get multi version start(ls_id={id:1}, tablet_id={id:50148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.881693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.881765] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.881780] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=18] success to update min reserved snapshot(reserved_snapshot=1736324515839366000, duration=1800, snapshot_gc_ts=1736326315839366000) [2025-01-08 00:52:02.881827] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.882042] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101038}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.882121] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.882575] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.882644] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.882724] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.882789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.882849] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.882908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101090}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.882968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.883063] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=22][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922109248 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:5990} write_epoch=1, prev=this=0x2ad9220fbfa0 trans_version={val:1736325662501492000, v:0} scn={val:1736325662501492000, v:0} tx_id={txid:22451} prev=0x2ad9220fbb30 next=(nil) modify_count=4 acc_checksum=4017978693 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:84060} write_epoch=3, snapshot_version={val:1736326322874106000, v:0}, *this={this=0x2ad9220b1d68 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fbfa0 latest_compact_node=(nil) max_trans_version={val:1736325662501492000, v:0} max_trans_id=22451 max_elr_trans_version={val:1736325662501492000, v:0} max_elr_trans_id=22451 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325662501492000, v:0} min_modify_scn={val:1736324403516713000, v:0}}) [2025-01-08 00:52:02.883105] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=81] get multi version start(ls_id={id:1}, tablet_id={id:60328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.883166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.883111] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=46][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922109248 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:5990} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326352778464, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326352778464, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326322874106000, v:0}, tx_id:{txid:23722}, scn:{branch:0, seq:5987}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:23722}, tx_desc:{this:0x2ad8b87444e0, tx_id:{txid:23722}, state:1, addr:"192.168.159.138:2882", tenant_id:1001, session_id:3221815713, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326322874106, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326322986179, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6b4a50, mem_ctx:0x2ad8fb6b5fd0, tx_scn:{branch:0, seq:5990}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:02.883304] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.883309] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=196][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:23722}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:5990}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:02.883333] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=22][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:23722} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:02.883365] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:02.883366] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324514412287000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324514412287000, max_merged_snapshot=1) [2025-01-08 00:52:02.883386] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:116) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E7D-0-0] [lt=15] SSTableGCTask(cost_ts=112312) [2025-01-08 00:52:02.883378] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0b4, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c0c4, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0d3, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0db, hex: B921B2F92D2B0600, int: 1736326322987449}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0e3, hex: A1679B5797AB1818, int: 1736326322849540001}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0eb, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:02.883515] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:50) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=50] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=5606, insert_dup_release_count=5606, col_map_alloc_count=0, col_map_release_count=0) [2025-01-08 00:52:02.883610] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=83][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:02.883737] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=8][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad9221093e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:5990} write_epoch=1, prev=this=0x2ad9220fc2d0 trans_version={val:1736325662501492000, v:0} scn={val:1736325662501492000, v:0} tx_id={txid:22451} prev=0x2ad9220fbc80 next=(nil) modify_count=4 acc_checksum=3572044856 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:84060} write_epoch=3, snapshot_version={val:1736326322874106000, v:0}, *this={this=0x2ad9220b1f20 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fc2d0 latest_compact_node=(nil) max_trans_version={val:1736325662501492000, v:0} max_trans_id=22451 max_elr_trans_version={val:1736325662501492000, v:0} max_elr_trans_id=22451 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325662501492000, v:0} min_modify_scn={val:1736324403516713000, v:0}}) [2025-01-08 00:52:02.883849] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=111][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad9221093e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:5990} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326352778464, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326352778464, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326322874106000, v:0}, tx_id:{txid:23722}, scn:{branch:0, seq:5987}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:23722}, tx_desc:{this:0x2ad8b87444e0, tx_id:{txid:23722}, state:1, addr:"192.168.159.138:2882", tenant_id:1001, session_id:3221815713, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326322874106, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326322986179, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6b4a50, mem_ctx:0x2ad8fb6b5fd0, tx_scn:{branch:0, seq:5990}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:02.884109] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=258][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:23722}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:5990}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:02.884210] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=99][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":2},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:23722} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:02.884400] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=188][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:02.884502] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=97][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c174, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c17c, hex: 0200000000000000, int: 2}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c184, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c193, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c19b, hex: B921B2F92D2B0600, int: 1736326322987449}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c1a3, hex: 909DD74097AB1818, int: 1736326322467610000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c1ab, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:02.884650] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=74][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:02.884693] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922109578 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:5990} write_epoch=1, prev=this=0x2ad9220fc138 trans_version={val:1736325662501492000, v:0} scn={val:1736325662501492000, v:0} tx_id={txid:22451} prev=0x2ad9220fbbd8 next=(nil) modify_count=4 acc_checksum=733862275 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:84060} write_epoch=3, snapshot_version={val:1736326322874106000, v:0}, *this={this=0x2ad9220b20d8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fc138 latest_compact_node=(nil) max_trans_version={val:1736325662501492000, v:0} max_trans_id=22451 max_elr_trans_version={val:1736325662501492000, v:0} max_elr_trans_id=22451 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325662501492000, v:0} min_modify_scn={val:1736324403516713000, v:0}}) [2025-01-08 00:52:02.884803] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=108][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922109578 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:5990} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326352778464, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326352778464, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326322874106000, v:0}, tx_id:{txid:23722}, scn:{branch:0, seq:5987}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:23722}, tx_desc:{this:0x2ad8b87444e0, tx_id:{txid:23722}, state:1, addr:"192.168.159.138:2882", tenant_id:1001, session_id:3221815713, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326322874106, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326322986179, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6b4a50, mem_ctx:0x2ad8fb6b5fd0, tx_scn:{branch:0, seq:5990}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:02.884948] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=143][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:23722}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:5990}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:02.884962] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=13][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:23722} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:02.885066] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=102][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:02.885079] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c234, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c23c, hex: 0300000000000000, int: 3}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c244, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c253, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c25b, hex: B921B2F92D2B0600, int: 1736326322987449}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c263, hex: A1679B5797AB1818, int: 1736326322849540001}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c26b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:02.885143] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=40][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:02.885245] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=8][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad922109710 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:5990} write_epoch=1, prev=this=0x2ad9220fbe08 trans_version={val:1736325662501492000, v:0} scn={val:1736325662501492000, v:0} tx_id={txid:22451} prev=0x2ad9220fba88 next=(nil) modify_count=4 acc_checksum=2892094923 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:84060} write_epoch=3, snapshot_version={val:1736326322874106000, v:0}, *this={this=0x2ad9220b2290 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fbe08 latest_compact_node=(nil) max_trans_version={val:1736325662501492000, v:0} max_trans_id=22451 max_elr_trans_version={val:1736325662501492000, v:0} max_elr_trans_id=22451 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325662501492000, v:0} min_modify_scn={val:1736324403516713000, v:0}}) [2025-01-08 00:52:02.885300] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=54][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad922109710 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:23722} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:5990} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326352778464, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326352778464, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326322874106000, v:0}, tx_id:{txid:23722}, scn:{branch:0, seq:5987}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:23722}, tx_desc:{this:0x2ad8b87444e0, tx_id:{txid:23722}, state:1, addr:"192.168.159.138:2882", tenant_id:1001, session_id:3221815713, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326322874106, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326322986179, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6b4a50, mem_ctx:0x2ad8fb6b5fd0, tx_scn:{branch:0, seq:5990}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:02.885368] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=68][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:23722}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:5990}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:02.885381] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=12][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":4},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:23722} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:02.885458] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=74][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:02.885468] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c2f4, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c2fc, hex: 0400000000000000, int: 4}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c304, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c313, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c31b, hex: B921B2F92D2B0600, int: 1736326322987449}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c323, hex: FFFFFFFFFFFFFF3F, int: 4611686018427387903}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c32b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:02.885532] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=39][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:02.891853] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:52:02.891882] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] try gc retain ctx [2025-01-08 00:52:02.891890] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:52:02.891918] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1736326322346465004, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326312645069001, v:0}, keep_alive_scn_:{val:1736326317629447001, v:0}, update_ts_:1736326322891100}) [2025-01-08 00:52:02.892783] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:02.892899] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326319418178002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326330087570000, latest_log_ts:{val:1736326313644270001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:02.893374] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=173] finish update min start scn(max_decided_scn={val:1736326322346465004, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326312645069001, v:0}, keep_alive_scn_:{val:1736326317629447001, v:0}, update_ts_:1736326322891100}) [2025-01-08 00:52:02.894291] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=24] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:02.894441] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326314412287000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:02.894450] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326317629447000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326319418178002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:02.894584] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=60] finish update min start scn(max_decided_scn={val:1736326322353812004, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326310862856000, v:0}, keep_alive_scn_:{val:1736326321344654003, v:0}, update_ts_:1736326322891100}) [2025-01-08 00:52:02.895317] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326317803522002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326319418178002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:02.895473] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=142] finish update min start scn(max_decided_scn={val:1736326322345307003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326314165163003, v:0}, keep_alive_scn_:{val:1736326319864970000, v:0}, update_ts_:1736326322891100}) [2025-01-08 00:52:02.896001] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:02.896116] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326317454295003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326319418178002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:02.896906] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.896965] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.896992] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.897323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:02.898171] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:02.900686] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84494932}], scn_array=[cnt:1, {val:1736326322827281001, v:0}], curr_size=112, accum_size=4221, time_cost=18735) [2025-01-08 00:52:02.901359] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:52:02.901495] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=16] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324515839366000) [2025-01-08 00:52:02.901525] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] update reserved snapshot finished(cost_ts=19748, reserved_snapshot=1736324515839366000) [2025-01-08 00:52:02.902182] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=15] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:25234}, to_seq_no={branch:0, seq:5988}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:23722} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:02.903939] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BED9-0-0] [lt=17] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:52:02.905053] INFO [STORAGE.TRANS] rollback_to_savepoint (ob_trans_part_ctx.cpp:8738) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=107] rollback succeed trivially(trans_id={txid:23722}, ls_id={id:1}, op_sn=3, to_scn={branch:0, seq:5988}, last_scn={branch:0, seq:5988}) [2025-01-08 00:52:02.908252] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2221599}], scn_array=[cnt:1, {val:1736326322838758000, v:0}], curr_size=113, accum_size=1089, time_cost=13377) [2025-01-08 00:52:02.908803] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18744419}], scn_array=[cnt:1, {val:1736326322838758000, v:0}], curr_size=113, accum_size=1512, time_cost=14303) [2025-01-08 00:52:02.914807] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2484) [46143][pnio1][T1001][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=6][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:23722}, sender_ls_id={id:1}, receiver_ls_id={id:9223372036854775807}, msg_type=67, status=-4018, receiver_addr="192.168.159.138:2882", request_id=1736324339343070) [2025-01-08 00:52:02.914957] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:288) [46143][pnio1][T0][YB42C0A89F8A-00062B2D1EC9BA0F-0-0] [lt=147][errcode=0] trx rpc callback(ret=0, status=-4018, dst="192.168.159.138:2882", result={status:-4018, send_timestamp:1736326322905091, private_data:{val:18446744073709551615, v:3}}) [2025-01-08 00:52:02.919073] WDIAG [SERVER] inner_get_next_row (ob_information_query_response_time.cpp:107) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D198BB258-0-0] [lt=19][errcode=-4008] fail to execute(ret=-4008) [2025-01-08 00:52:02.948887] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=40] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:02.950605] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=36] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:02.950771] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326315839366000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:02.958333] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:02.959820] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC8-0-0] [lt=113][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:02.962045] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC8-0-0] [lt=112][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:02.966986] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A491F-0-0] [lt=60] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:02.967024] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A491F-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:02.967042] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A491F-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:02.968403] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2288, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:02.968430] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:02.968450] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=170, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:02.968527] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=67, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:02.982658] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=10] BlockGCTimerTask success(ret=0, cost_time_us=62, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:52:02.986647] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF73-0-0] [lt=10] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326323095510, keepalive_ts:-1}) [2025-01-08 00:52:02.986679] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF73-0-0] [lt=30] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326323095510, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:02.992244] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=111, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:02.992301] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:02.992496] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=110, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:02.992550] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:52:02.992576] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:52:02.992591] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:02.994691] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3855, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:02.994717] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1634) [2025-01-08 00:52:02.994737] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=591, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:02.994796] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1634) [2025-01-08 00:52:02.994818] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:02.995367] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=119] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=4343, avg_group_log_size=334) [2025-01-08 00:52:02.995649] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3731, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:02.995672] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:52:02.995686] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=587, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:02.995784] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:52:02.995819] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=44, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:02.996722] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9706-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:02.996741] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9706-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8419, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:02.996817] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=84] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:02.996903] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=110] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4343) [2025-01-08 00:52:02.997385] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9707-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:02.997450] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9707-0-0] [lt=63] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7002, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:02.997462] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9707-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=134, avg_log_submit_to_flush_time=4515, avg_log_submit_to_slide_time=6868) [2025-01-08 00:52:02.997530] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1634, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:03.000123] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BBF-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:03.000686] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58661][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DA8-0-0] [lt=49] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=124, avg_log_gen_to_submit_time=221, avg_log_submit_to_flush_time=4584, avg_log_submit_to_slide_time=7283) [2025-01-08 00:52:03.000921] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4343, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:03.006719] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCE-0-0] [lt=45] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326323116813, keepalive_ts:-1}) [2025-01-08 00:52:03.006769] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCE-0-0] [lt=48] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326323116813, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:03.027023] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=81] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=85, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:03.027083] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=11812, avg_group_log_size=694) [2025-01-08 00:52:03.029038] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=3004, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:03.029110] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=115, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:03.029148] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=55] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11812) [2025-01-08 00:52:03.029167] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=37, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:03.031754] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F89-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=23, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:03.031794] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F89-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=5871, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:03.031805] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2F89-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=81, avg_log_gen_to_submit_time=180, avg_log_submit_to_flush_time=3472, avg_log_submit_to_slide_time=5690) [2025-01-08 00:52:03.032021] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=44] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=11812, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:03.032583] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326322345307003, v:0}, new_max_version={val:1736326322345307003, v:0}, last_min_version={val:1736326322345307002, v:0}, last_max_version={val:1736326322345307002, v:0}, total_time_us=7782, delta_us=687274) [2025-01-08 00:52:03.032632] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326322345307003, v:0}, min_version={val:1736326322345307003, v:0}, max_version={val:1736326322345307003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:03.045901] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=103] runTimer to refresh locality_info(ret=0) [2025-01-08 00:52:03.049141] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:346) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=327] set_server_locality_array success(server_locality_array=[{inited:true, addr:"192.168.159.137:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322580423131, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.138:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581270010, server_stop_time:0, server_status:1}, {inited:true, addr:"192.168.159.139:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1736322581175973, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-01-08 00:52:03.049272] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=40] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-01-08 00:52:03.049293] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=19] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-01-08 00:52:03.049312] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=12] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1001, primary_zone=RANDOM) [2025-01-08 00:52:03.049321] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] add_locality_zone success(locality_zone={tenant_id:1001, region_priority:18446744073709551615}) [2025-01-08 00:52:03.049340] INFO [SHARE] load_region (ob_locality_table_operator.cpp:336) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1002, primary_zone=RANDOM) [2025-01-08 00:52:03.049350] INFO [SHARE] load_region (ob_locality_table_operator.cpp:363) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=10] add_locality_zone success(locality_zone={tenant_id:1002, region_priority:18446744073709551615}) [2025-01-08 00:52:03.049364] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [46194][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=10] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1001, region_priority:18446744073709551615}, {tenant_id:1002, region_priority:18446744073709551615}]}) [2025-01-08 00:52:03.056784] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852BF-0-0] [lt=40] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326318396767, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:52:03.056892] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852BF-0-0] [lt=104] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326318398655, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:52:03.056927] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1230) [46099][TimerWK1][T0][YB42C0A89F89-00062B2D172852BF-0-0] [lt=33] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, renew_time:1736326318400448, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, renew_time:1736326318400449, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, renew_time:1736326318400451, replica_locations:[{server:"192.168.159.137:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, renew_time:1736326318400444, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"192.168.159.138:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"192.168.159.139:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-01-08 00:52:03.058361] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.063723] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1001, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18254, max_lsn:2221834, max_scn:{val:1736326322838758001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2221834}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2221712}, last_submit_end_lsn:{lsn:2221834}, last_submit_log_id:18254, last_submit_log_pid:1, max_flushed_lsn:{lsn:2221712}, max_flushed_end_lsn:{lsn:2221834}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2221834}, last_slide_log_id:18254, last_slide_scn:{val:1736326322838758001, v:0}, last_slide_lsn:{lsn:2221712}, last_slide_end_lsn:{lsn:2221834}, last_slide_log_pid:1, last_slide_log_accum_checksum:1491295459, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924f6a530}, [StateMgr]={this:0x2ad924f6e830, self:"192.168.159.137:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18255, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324447115984, palf_role_change_cb:0x2ad924f752f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924f6bc70}, [ModeMgr]={palf_id:1001, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324447762546, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436756804001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436756804000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/meta", dir_fd:177, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924f70a38, flashback_version:0}, log_storage:{log_tail:{lsn:2221834}, readable_log_tail:{lsn:2221834}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64882934, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1736324447114985, this:0x2ad924f6ef70}, [Reconfirm]={palf_id:1001, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:1"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324447117136, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924f6e9b0}) [2025-01-08 00:52:03.064087] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=354] [PALF_DUMP](palf_id=1001, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326322995703, last_flushed_end_lsn:{lsn:2221834}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326322996515, last_flushed_end_lsn:{lsn:2221834}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326322997012, last_flushed_end_lsn:{lsn:2221834}}]) [2025-01-08 00:52:03.064113] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] current log_task status(palf_id=1001, self="192.168.159.137:2882", log_id=18255, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326322995703, last_flushed_end_lsn:{lsn:2221834}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326322996515, last_flushed_end_lsn:{lsn:2221834}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326322997012, last_flushed_end_lsn:{lsn:2221834}}]) [2025-01-08 00:52:03.070942] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB3F-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:03.070968] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB3F-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14494, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:03.070979] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB3F-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=27, avg_log_submit_to_flush_time=2541, avg_log_submit_to_slide_time=14466) [2025-01-08 00:52:03.071079] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:03.071114] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18441, miss_cnt:0, cache_read_size:2255967, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:03.071175] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=159, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:03.071192] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=16] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=8, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:03.073438] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=12401) [2025-01-08 00:52:03.081165] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=72] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:52:03.096644] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=17] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:03.099949] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BC1-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:03.099992] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BC1-0-0] [lt=32] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:03.100023] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BC1-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=24, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:03.100033] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BC1-0-0] [lt=9] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=35950, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:03.100041] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BC1-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=2847, avg_log_submit_to_slide_time=35936) [2025-01-08 00:52:03.100166] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:03.100293] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18340, miss_cnt:0, cache_read_size:2243315, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:03.101492] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2490, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:03.101519] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:03.101538] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=194, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:03.101630] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=67, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:03.125229] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF0-0-0] [lt=29] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:52:03.125288] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF0-0-0] [lt=49] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=129) [2025-01-08 00:52:03.135952] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=79] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:03.153073] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=45] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:03.153687] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:52:03.153827] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%", tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:52:03.154618] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=31] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:03.154712] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326315701616000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:03.155487] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0D7-0-0] [lt=65][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:03.155730] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] [COMPACTION DAG STATUS] (start_time=1736326312828346, end_time=1736326323155728, dag_status={}) [2025-01-08 00:52:03.155757] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=24] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:03.155770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:03.155778] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:03.155785] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:03.155792] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:03.155799] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:03.155806] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:03.155813] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:03.155819] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:03.155827] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155837] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155846] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155854] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155862] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155870] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155878] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155886] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155894] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155902] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155910] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155918] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155926] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155934] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155942] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.155995] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156008] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156016] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156025] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156033] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156041] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156049] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156057] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156065] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156073] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156080] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156088] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156096] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156104] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156112] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156215] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=101] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156227] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156235] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156243] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156251] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156259] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156267] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156275] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156283] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156291] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156299] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156307] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156315] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156323] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156331] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156540] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=207] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156562] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156584] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156593] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156601] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156609] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156617] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156625] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156633] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156641] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156649] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156657] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156665] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156673] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156681] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156688] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156696] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156704] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156712] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156728] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156736] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156793] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=56] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156802] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156810] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156818] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156826] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156834] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:03.156843] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:52:03.156851] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:52:03.156858] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:52:03.156863] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:52:03.156869] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:52:03.156874] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:52:03.156879] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:52:03.156884] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:52:03.156889] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:52:03.156894] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:52:03.156901] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:52:03.157027] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=18] gc stale ls task succ [2025-01-08 00:52:03.157106] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1003}, {id:1001}, {id:1}]) [2025-01-08 00:52:03.158535] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:03.158556] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:03.158852] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.163274] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB871A2-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:03.163319] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:03.163331] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:03.166161] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:03.166255] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:52:03.168442] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=129] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:52:03.168487] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=35] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=91) [2025-01-08 00:52:03.181776] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185455-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:03.182128] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:03.182153] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-01-08 00:52:03.182264] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=108] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:03.182296] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:03.182310] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193827093331, flush_io_finished_ret_:0, flush_io_finished_round_:1981, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:03.224788] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46385][T1_L0_G0][T1][YB42C0A89F8B-00062B2D171BBD9A-0-0] [lt=39] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.779975576400756836e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=889, Throttle Trigger(MB)=533, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.837070840316183862e+01) [2025-01-08 00:52:03.225025] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46385][T1_L0_G0][T1][YB42C0A89F8B-00062B2D171BBD9A-0-0] [lt=56] get number(ret=0, service_type_=0, range=1, base_id=1736326323224494000, start_id=1736326323224494000, end_id=1736326323224494001) [2025-01-08 00:52:03.230034] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:03.230172] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=82, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:03.230637] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DAB-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4889) [2025-01-08 00:52:03.233877] INFO [LIB] stat (utility.h:1210) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DAC-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=44, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:03.233917] INFO [LIB] stat (utility.h:1210) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DAC-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=7179, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:03.235798] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46101][TimerWK3][T0][YB42C0A89F89-00062B2D17F852FB-0-0] [lt=34] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:52:03.249692] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46503][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D6-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193828163523, cache_obj->added_lc()=false, cache_obj->get_object_id()=13703, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:03.250968] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D7-0-0] [lt=25][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:03.252900] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857D9-0-0] [lt=24][errcode=-4201] hashtable not init [2025-01-08 00:52:03.260266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.268384] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:52:03.269919] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=103] ====== traversal_flush timer task ====== [2025-01-08 00:52:03.270099] INFO [MDS] flush (mds_table_mgr.cpp:195) [46305][T1_TimerWK1][T1][YB42C0A89F89-00062B2D1768578D-0-0] [lt=23] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326323227770000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339254996000, latest_log_ts:{val:1736326322625057001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:03.270256] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46305][T1_TimerWK1][T1][YB42C0A89F89-00062B2D1768578D-0-0] [lt=149] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:52:03.270696] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=102] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1001, min_version={val:1736326322860272000, v:0}, base_version={val:1736326322815770000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326323070327000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326323270674, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326322860272000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326323199780, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326323070327000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326323218746, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:52:03.271423] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58661][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=47] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:52:03.273366] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=106] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:03.281859] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=105] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:52:03.283650] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2025-01-08 00:52:03.283679] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:52:03.292159] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A86-0-0] [lt=35] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326323292142}) [2025-01-08 00:52:03.297448] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=80] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:03.299302] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=58] ====== checkpoint timer task ====== [2025-01-08 00:52:03.299315] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BA-0-0] [lt=32] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326323299161, sql_renew_time:1736326323299161, is_mark_fail:false}]) [2025-01-08 00:52:03.299354] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BA-0-0] [lt=38] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326323292142}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326323299161, sql_renew_time:1736326323299161, is_mark_fail:false}]) [2025-01-08 00:52:03.300065] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:03.300109] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=44] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:03.300179] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326319418178002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326330087570000, latest_log_ts:{val:1736326313644270001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:03.300810] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=626] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:52:03.300907] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=90] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:52:03.300924] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=0, rec_log_ts={val:1736326313644270001, v:0}) [2025-01-08 00:52:03.300959] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=34] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:52:03.300976] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:03.300995] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:03.301009] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:52:03.301056] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:03.301123] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=65] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:52:03.302028] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:03.302060] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=32] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:03.302100] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=16] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326317629447000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326319418178002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:03.302287] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=184] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:52:03.302304] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:52:03.302320] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:52:03.302336] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:52:03.302353] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2222078}, scn={val:1736326322838758003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:52:03.302720] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BC5-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18432, log_task={header:{begin_lsn:{lsn:2243437}, end_lsn:{lsn:2243559}, log_id:18432, min_scn:{val:1736326322849540003, v:0}, max_scn:{val:1736326322849540003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2243315}, prev_proposal_id:1, committed_end_lsn:{lsn:2243437}, data_checksum:940009244, accum_checksum:3394639832, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326323200311, freeze_ts:1736326323200311, submit_ts:1736326323200344, flushed_ts:1736326323205043, gen_to_freeze cost time:0, gen_to_submit cost time:33, submit_to_flush cost time:4699}, fs_cb_begin_ts=1736326323302708, log_life_time=102397) [2025-01-08 00:52:03.310158] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBB1-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193828223990, cache_obj->added_lc()=false, cache_obj->get_object_id()=13704, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:03.310809] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=53] ====== check clog disk timer task ====== [2025-01-08 00:52:03.310839] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:52:03.312819] INFO [PALF] pread (log_reader.cpp:130) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=5964) [2025-01-08 00:52:03.316931] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8843, offset=2512625}, end_cursor:ObLogCursor{file_id=1, log_id=8844, offset=2512922}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d24050, len:2319, log_data_len:297, seq:8843, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:03.317297] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326322838758003, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:52:03.317336] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:03.317362] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:52:03.318037] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:03.318076] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=38] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:03.318127] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326317803522002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326319418178002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:03.318253] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=123] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:52:03.318271] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:52:03.318290] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:52:03.318307] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:52:03.318322] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2256089}, scn={val:1736326322860272003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:52:03.319286] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.319325] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.322095] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=47] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8844, offset=2512922}, end_cursor:ObLogCursor{file_id=1, log_id=8845, offset=2513219}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d28050, len:2022, log_data_len:297, seq:8844, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:03.322297] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326322860272003, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:52:03.322331] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:03.322353] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:52:03.323142] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:03.323178] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=35] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:03.323272] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=69] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326317454295003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326319418178002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:03.323340] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=65] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:52:03.323354] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:52:03.323369] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:52:03.323383] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:52:03.323397] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2243681}, scn={val:1736326322849540004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:52:03.324980] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=102] start to print blacklist info [2025-01-08 00:52:03.325429] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=54] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8845, offset=2513219}, end_cursor:ObLogCursor{file_id=1, log_id=8846, offset=2513516}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d2c050, len:1725, log_data_len:297, seq:8845, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:03.325496] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326322849540004, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:52:03.325523] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:03.325567] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:52:03.325581] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:52:03.325625] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=53] ls blacklist refresh finish(cost_time=34922) [2025-01-08 00:52:03.326215] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=94] PNIO [ratelimit] time: 1736326323326211, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007551, add_bytes: 0 [2025-01-08 00:52:03.329000] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=27] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8846, offset=2513516}, end_cursor:ObLogCursor{file_id=1, log_id=8847, offset=2513821}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d30050, len:1428, log_data_len:305, seq:8846, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:03.329443] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7B-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:03.329551] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7B-0-0] [lt=48] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326322838758004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326322838758004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:03.333702] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:03.334615] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:491) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E7B-0-0] [lt=22] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=1, MAX_GAP=900000000000, new_sync_scn={val:1736326322022412002, v:0}, new_replayable_scn={val:1736326322022412002, v:0}, new_readable_scn={val:1736326322022412001, v:0}, old_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326320841974004, replayable_scn:1736326320841974004, readable_scn:1736326320841974004, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}) [2025-01-08 00:52:03.339520] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8847, offset=2513821}, end_cursor:ObLogCursor{file_id=1, log_id=8848, offset=2514118}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d38050, len:1123, log_data_len:297, seq:8847, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:03.343612] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE973-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326323338688000, start_id=1736326323338688000, end_id=1736326323338688001) [2025-01-08 00:52:03.345054] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7B-0-0] [lt=50] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:03.345118] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7B-0-0] [lt=43] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326322838758004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326322838758004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:03.349234] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:03.352318] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8848, offset=2514118}, end_cursor:ObLogCursor{file_id=1, log_id=8849, offset=2514415}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d34050, len:826, log_data_len:297, seq:8848, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:03.354082] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8849, offset=2514415}, end_cursor:ObLogCursor{file_id=1, log_id=8850, offset=2514712}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d3c050, len:529, log_data_len:297, seq:8849, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:03.355567] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:03.360409] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.369474] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A79-0-0] [lt=90] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:03.380798] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:03.380916] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:03.395914] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:52:03.395951] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:52:03.399575] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:03.399627] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:14, bw:65034, [delay/us]:prepare:6, schedule:80, submit:116, rt:2966, total:3170) [2025-01-08 00:52:03.399642] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:03.399708] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=51] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:14, avg_bw:65034, [delay/us]: prepare:6, schedule:80, submit:116, device:2966, total:3170) [2025-01-08 00:52:03.399814] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:03.399916] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:73159, [delay/us]:prepare:8, schedule:124, submit:176, rt:2372, total:2680) [2025-01-08 00:52:03.399932] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:03.399986] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:73166, [delay/us]: prepare:8, schedule:124, submit:176, device:2372, total:2680) [2025-01-08 00:52:03.400001] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:03.400064] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:7, bw:28453, [delay/us]:prepare:5, schedule:53, submit:1, rt:1421, total:1481) [2025-01-08 00:52:03.400090] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:2, hold_mem:16384 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:39, bw:158528, [delay/us]:prepare:6, schedule:191, submit:194, rt:2824, total:3216) [2025-01-08 00:52:03.400072] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:03.400099] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12194, [delay/us]:prepare:1, schedule:105, submit:33, rt:393, total:534) [2025-01-08 00:52:03.400109] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=6, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49998, free_result_cnt=49998, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:03.400286] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=170] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:7, avg_bw:28459, [delay/us]: prepare:5, schedule:53, submit:1, device:1421, total:1481) [2025-01-08 00:52:03.400304] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12197, [delay/us]: prepare:1, schedule:105, submit:33, device:393, total:534) [2025-01-08 00:52:03.400312] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:39, avg_bw:158562, [delay/us]: prepare:6, schedule:191, submit:194, device:2824, total:3216) [2025-01-08 00:52:03.400339] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:03.400362] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4065, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:719, total:728) [2025-01-08 00:52:03.400388] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:03.400418] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4065, [delay/us]: prepare:0, schedule:0, submit:600000000, device:719, total:728) [2025-01-08 00:52:03.400492] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:03.400572] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:03.400583] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:03.400591] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:03.400599] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=281kB/s, net_out=246kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:03.404470] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:03.404832] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4054, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:03.405376] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=84] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=510, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:03.405892] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=73] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=38, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:03.427124] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11813) [2025-01-08 00:52:03.430103] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4889) [2025-01-08 00:52:03.433767] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:03.450716] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBB3-0-0] [lt=90][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193828364547, cache_obj->added_lc()=false, cache_obj->get_object_id()=13705, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:03.452296] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=132] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:52:03.453335] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58604][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=14] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:52:03.458030] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5330949543, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:03.458066] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:03.458078] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad90857ded0) [2025-01-08 00:52:03.458087] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5330949543, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:03.459571] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.459641] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.460650] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=97][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.464241] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=440] ====== tenant freeze timer task ====== [2025-01-08 00:52:03.464460] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] ====== tenant freeze timer task ====== [2025-01-08 00:52:03.464482] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70704720, tenant_id=1001) [2025-01-08 00:52:03.464503] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:52:03.468851] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:344K/82:195661 disk:344K/82:195661 t1m2:60K/14:237223 t1m2g0:60K/14:237223 t1002m2:204K/51:206200 t1002m2g0:204K/51:206200 t1001m2:80K/17:129178 t1001m2g0:80K/17:129178 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:03.482056] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:52:03.497821] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:03.500281] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:52:03.500620] WDIAG read_multi_row (ob_table_access_helper.h:238) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=164][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1001, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:52:03.502509] INFO delete_row (ob_table_access_helper.h:339) [50584][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=44] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1001, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326323500666) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326323500666) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:52:03.506416] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:52:03.508646] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58615][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=46] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:52:03.519297] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=43] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.536924] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] clear dtl interm result cost(us)(clear_cost=7669, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:52:03.542369] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=48] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.543544] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.560818] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.579564] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB624-0-0] [lt=124] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:03.579622] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB624-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326323338688000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326323338688000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:03.583408] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=49] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:52:03.596009] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB624-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1736326323593695000, start_id=1736326323593695000, end_id=1736326323593695001) [2025-01-08 00:52:03.596645] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:52:03.596746] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%", tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-01-08 00:52:03.601373] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=27] rpc statistics(total_trans_msg_count=5, total_batch_msg_count=0) [2025-01-08 00:52:03.601958] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CE-0-0] [lt=86] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:03.602011] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CE-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326323355585002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326323355585002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:03.661073] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.677234] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO [ratelimit] time: 1736326323677230, bytes: 886135662, bw: 0.307476 MB/s, add_ts: 1001498, add_bytes: 322895 [2025-01-08 00:52:03.682320] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:52:03.698761] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=188] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:03.703167] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=94] current monitor number(seq_=-1) [2025-01-08 00:52:03.713653] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D55-0-0] [lt=34] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:52:03.726312] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:194) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=47] get wrs ts(ls_id={id:1}, delta=375792, log_service_ts=1736326323348583, min_tx_service_ts=4611686018427387, timestamp={val:1736326323348583002, v:0}) [2025-01-08 00:52:03.726367] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=48] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=8, Other_Error_Cnt=0, Submit_Succ_Cnt=75, last_scn="{val:1736326322838758000, v:0}", last_lsn={lsn:18744475}, last_gts={val:1736326323348583000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-01-08 00:52:03.735242] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1609BCC9-0-0] [lt=51] submit log success(service_type_=0, last_id=1736326340087570000, limited_id=1736326340087570000) [2025-01-08 00:52:03.735308] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50913][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1609BCC9-0-0] [lt=65] submit log with lock success(service_type_=0, last_id=1736326340087570000, limited_id=1736326340087570000) [2025-01-08 00:52:03.743176] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=10] update id meta(ret=0, service_type=0, limited_id=1736326340087570000, latest_scn={val:1736326323734680001, v:0}, *this={tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326320841974004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326340087570000, latest_log_ts:{val:1736326323734680001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:52:03.743307] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=129] handle submit callback(service_type_=0, success=true, used_ts=8126, log_ts={val:1736326323734680001, v:0}, limited_id=1736326340087570000) [2025-01-08 00:52:03.743321] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=14] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:52:03.743331] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=8] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:52:03.743344] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=11] [APPLY STAT CB APPEND COST TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=172, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:52:03.743365] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=20] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=44415, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:52:03.743373] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:52:03.743380] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=-2881, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:52:03.743387] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=18, stat_interval=5000000, avg (count/cost)=41705, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:52:03.761368] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.771835] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A492F-0-0] [lt=128][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18540, log_task={header:{begin_lsn:{lsn:2256577}, end_lsn:{lsn:2256699}, log_id:18540, min_scn:{val:1736326323355585003, v:0}, max_scn:{val:1736326323355585003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2256455}, prev_proposal_id:1, committed_end_lsn:{lsn:2256577}, data_checksum:1715074492, accum_checksum:2852935447, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326323670304, freeze_ts:1736326323670304, submit_ts:1736326323670318, flushed_ts:1736326323674399, gen_to_freeze cost time:0, gen_to_submit cost time:14, submit_to_flush cost time:4081}, fs_cb_begin_ts=1736326323771806, log_life_time=101502) [2025-01-08 00:52:03.779551] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=25] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:52:03.785453] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] BlockGCTimerTask success(ret=0, cost_time_us=17, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:52:03.799352] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708517B-0-0] [lt=68] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:03.835605] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:03.838292] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1736326323338688004, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3, epoch_tstamp:1736326323837980}, version_delta=499602) [2025-01-08 00:52:03.851121] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58679][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:03.851120] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:52:03.854532] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=21] gc stale ls task succ [2025-01-08 00:52:03.854779] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=89] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:03.854811] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-01-08 00:52:03.854822] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:03.854836] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:03.854850] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193827768462, flush_io_finished_ret_:0, flush_io_finished_round_:1986, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:03.855720] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=31] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:03.858634] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:03.858751] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15223489809, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:03.858834] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=21] recycle one thread(this=0x2ad8881fc3b0, total=5, remain=4) [2025-01-08 00:52:03.858889] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=54] inc thread count(*this={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15223489809, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-01-08 00:52:03.860709] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=17][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad92210a448 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43669} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:1943} write_epoch=1, prev=this=0x2ad9220fd230 trans_version={val:1736325663449545000, v:0} scn={val:1736325663449545000, v:0} tx_id={txid:42407} prev=0x2ad9220fcbe0 next=(nil) modify_count=4 acc_checksum=2345117193 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:84492} write_epoch=3, snapshot_version={val:1736326323854483000, v:0}, *this={this=0x2ad9220b4348 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fd230 latest_compact_node=(nil) max_trans_version={val:1736325663449545000, v:0} max_trans_id=42407 max_elr_trans_version={val:1736325663449545000, v:0} max_elr_trans_id=42407 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325663449545000, v:0} min_modify_scn={val:1736324403594508000, v:0}}) [2025-01-08 00:52:03.860785] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=74][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad92210a448 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43669} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:1943} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326353756656, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326353756656, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326323854483000, v:0}, tx_id:{txid:43669}, scn:{branch:0, seq:1940}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:43669}, tx_desc:{this:0x2ad8b878f2c0, tx_id:{txid:43669}, state:1, addr:"192.168.159.139:2882", tenant_id:1001, session_id:3222077594, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326323859459, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326323958243, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6c14d0, mem_ctx:0x2ad8fb6c2a50, tx_scn:{branch:0, seq:1943}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:03.860902] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=115][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:43669}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:1943}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:03.860923] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:43669} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:03.860952] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:03.860963] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0b4, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c0c4, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0d3, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0db, hex: 82C3BFF92D2B0600, int: 1736326323880834}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0e3, hex: 20F7649297AB1818, int: 1736326323835828000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c0eb, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:03.861055] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=49][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:03.861256] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=105][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad92210a5e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43669} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:1943} write_epoch=1, prev=this=0x2ad9220fcd68 trans_version={val:1736325663449545000, v:0} scn={val:1736325663449545000, v:0} tx_id={txid:42407} prev=0x2ad9220fc9e8 next=(nil) modify_count=4 acc_checksum=4084500327 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:84492} write_epoch=3, snapshot_version={val:1736326323854483000, v:0}, *this={this=0x2ad9220b4500 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fcd68 latest_compact_node=(nil) max_trans_version={val:1736325663449545000, v:0} max_trans_id=42407 max_elr_trans_version={val:1736325663449545000, v:0} max_elr_trans_id=42407 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325663449545000, v:0} min_modify_scn={val:1736324403594508000, v:0}}) [2025-01-08 00:52:03.861297] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=41][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad92210a5e0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43669} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:1943} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326353756656, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326353756656, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326323854483000, v:0}, tx_id:{txid:43669}, scn:{branch:0, seq:1940}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:43669}, tx_desc:{this:0x2ad8b878f2c0, tx_id:{txid:43669}, state:1, addr:"192.168.159.139:2882", tenant_id:1001, session_id:3222077594, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326323859459, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326323958243, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6c14d0, mem_ctx:0x2ad8fb6c2a50, tx_scn:{branch:0, seq:1943}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:03.861416] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=118][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:43669}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:1943}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:03.861430] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=13][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":2},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:43669} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:03.861451] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=19][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:03.861459] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=5][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c174, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c17c, hex: 0200000000000000, int: 2}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c184, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c193, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c19b, hex: 82C3BFF92D2B0600, int: 1736326323880834}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c1a3, hex: 68B99D7C97AB1818, int: 1736326323470449000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c1ab, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:03.861543] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=60][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:03.861495] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.861579] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=13][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad92210a778 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43669} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:1943} write_epoch=1, prev=this=0x2ad9220fd098 trans_version={val:1736325663449545000, v:0} scn={val:1736325663449545000, v:0} tx_id={txid:42407} prev=0x2ad9220fcb38 next=(nil) modify_count=4 acc_checksum=2387015914 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:84492} write_epoch=3, snapshot_version={val:1736326323854483000, v:0}, *this={this=0x2ad9220b46b8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fd098 latest_compact_node=(nil) max_trans_version={val:1736325663449545000, v:0} max_trans_id=42407 max_elr_trans_version={val:1736325663449545000, v:0} max_elr_trans_id=42407 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325663449545000, v:0} min_modify_scn={val:1736324403594508000, v:0}}) [2025-01-08 00:52:03.861679] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=98][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad92210a778 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43669} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:1943} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326353756656, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326353756656, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326323854483000, v:0}, tx_id:{txid:43669}, scn:{branch:0, seq:1940}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:43669}, tx_desc:{this:0x2ad8b878f2c0, tx_id:{txid:43669}, state:1, addr:"192.168.159.139:2882", tenant_id:1001, session_id:3222077594, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326323859459, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326323958243, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6c14d0, mem_ctx:0x2ad8fb6c2a50, tx_scn:{branch:0, seq:1943}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:03.861826] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=146][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:43669}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:1943}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:03.861896] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=69][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:43669} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:03.861920] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=23][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:03.861929] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c234, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c23c, hex: 0300000000000000, int: 3}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c244, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c253, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c25b, hex: 82C3BFF92D2B0600, int: 1736326323880834}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c263, hex: 4887C19197AB1818, int: 1736326323825117000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c26b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:03.861990] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=39][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:03.862069] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad92210a910 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43669} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:1943} write_epoch=1, prev=this=0x2ad9220fcf00 trans_version={val:1736325663449545000, v:0} scn={val:1736325663449545000, v:0} tx_id={txid:42407} prev=0x2ad9220fca90 next=(nil) modify_count=4 acc_checksum=1317628291 version=1736324394708351 type=0 flag=12 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:84492} write_epoch=3, snapshot_version={val:1736326323854483000, v:0}, *this={this=0x2ad9220b6770 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad9220fcf00 latest_compact_node=(nil) max_trans_version={val:1736325663449545000, v:0} max_trans_id=42407 max_elr_trans_version={val:1736325663449545000, v:0} max_elr_trans_id=42407 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736325663449545000, v:0} min_modify_scn={val:1736324403594508000, v:0}}) [2025-01-08 00:52:03.862103] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=33][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad92210a910 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:43669} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736324394708351 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:67} seq_no={branch:0, seq:1943} write_epoch=1, ctx={this:0x2ad90c1cf678, ls_id:{id:1}, ls:0x2ad908da4150, branch:0, timeout:1736326353756656, tablet_id:{id:444}, table_iter:0x2ad90c1cdb68, table_version:1736324367028792, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326353756656, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326323854483000, v:0}, tx_id:{txid:43669}, scn:{branch:0, seq:1940}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad908dafb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:43669}, tx_desc:{this:0x2ad8b878f2c0, tx_id:{txid:43669}, state:1, addr:"192.168.159.139:2882", tenant_id:1001, session_id:3222077594, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326323859459, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326323958243, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad8fb6c14d0, mem_ctx:0x2ad8fb6c2a50, tx_scn:{branch:0, seq:1943}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:03.862261] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=157][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:43669}, data:0x2ad90c1cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736324394708351, seq_no:{branch:0, seq:1943}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-01-08 00:52:03.862275] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"BIGINT UNSIGNED":4},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:43669} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:03.862297] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=20][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:03.862305] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2ad93ca0c640, local_datums:0x2ad93ca0c640}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c2f4, hex: E903000000000000, int: 1001}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c2fc, hex: 0400000000000000, int: 4}, col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x2ad93ca0c304, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c313, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c31b, hex: 82C3BFF92D2B0600, int: 1736326323880834}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c323, hex: FFFFFFFFFFFFFF3F, int: 4611686018427387903}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93ca0c32b, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:03.862620] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=217][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:03.862740] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=16] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"192.168.159.139:45062"}) [2025-01-08 00:52:03.863743] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB031-0-0] [lt=36] get number(ret=0, service_type_=0, range=1, base_id=1736326323862632000, start_id=1736326323862632000, end_id=1736326323862632001) [2025-01-08 00:52:03.864919] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1F3BAD77-0-0] [lt=35] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:1946}, to_seq_no={branch:0, seq:1941}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736324367028792 max_table_version=1736324367028792 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:43669} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:03.874332] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:03.874718] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE1-0-0] [lt=12] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:03.874747] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE1-0-0] [lt=26] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:03.878257] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:52:03.878306] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=12][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:52:03.878334] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=8][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:52:03.878340] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC440-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:03.881315] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:03.881381] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:03.883254] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:52:03.891601] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=42] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:03.891625] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=22] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:52:03.891674] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84996-0-0] [lt=17] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:03.891693] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84996-0-0] [lt=17] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:03.895739] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:263) [50687][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D2008458E-0-0] [lt=80] ObArchiveSender is running(thread_index=4) [2025-01-08 00:52:03.896976] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.897018] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.897043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.897312] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:03.899784] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:03.901709] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=46] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1478479, clean_num=31457) [2025-01-08 00:52:03.902346] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:52:03.906455] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:03.919750] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D160BB031-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1736326323919221000, start_id=1736326323919221000, end_id=1736326323919221001) [2025-01-08 00:52:03.935785] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:801) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185141-0-0] [lt=27] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, location:{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, location:{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}]) [2025-01-08 00:52:03.961653] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=96][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:03.972367] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=82] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43670734}, readable_log_tail:{lsn:43670734}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23434034, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:03.984559] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=48] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:52:03.984927] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4015] clock out of order(cur_ts=1736326323984925, cur_ts_=1736326323962900, delta=22025) [2025-01-08 00:52:03.985307] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4931-0-0] [lt=67] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:03.985682] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CC9-0-0] [lt=118][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:03.987383] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CC9-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:03.992112] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2592, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:03.992143] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:52:03.992165] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=436, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:03.992356] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=97, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:03.997809] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=76, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:03.997879] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:03.998032] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=39, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:03.998068] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:03.998087] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:52:03.998100] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:04.000761] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4085, avg_group_log_size=371) [2025-01-08 00:52:04.001905] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58670][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BD3-0-0] [lt=98] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:04.002872] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9725-0-0] [lt=52] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1348) [2025-01-08 00:52:04.003922] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4743, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:04.003949] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1348) [2025-01-08 00:52:04.003969] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=497, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:04.004073] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9726-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:04.004024] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4085) [2025-01-08 00:52:04.004231] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=26, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:04.004388] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=154] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7762, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:04.004408] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=104, avg_log_submit_to_flush_time=5294, avg_log_submit_to_slide_time=7658) [2025-01-08 00:52:04.004432] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:04.004504] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1348, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:04.004765] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:52:04.003465] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58680][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:04.006006] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A4B-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=86, avg_log_submit_to_flush_time=3636, avg_log_submit_to_slide_time=5476) [2025-01-08 00:52:04.006157] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4756, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:04.006226] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=68] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:04.006243] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=542, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:04.006289] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:04.006303] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8055, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:04.006363] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=41, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:04.006406] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4085, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:04.006445] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=90] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:04.019965] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58679][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DB3-0-0] [lt=29] init point thread id with(&point=0x5572d50e7700, idx_=1335, point=[thread id=58679, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58679) [2025-01-08 00:52:04.028806] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=227] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=128, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:04.028923] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=91] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=11683, avg_group_log_size=730) [2025-01-08 00:52:04.030441] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=51] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2379, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:04.030500] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=164, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:04.030549] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=11684) [2025-01-08 00:52:04.030604] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=49] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=24, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:04.032559] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3C31-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:04.032598] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3C31-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4761, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:04.032611] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3C31-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=117, avg_log_gen_to_submit_time=263, avg_log_submit_to_flush_time=2791, avg_log_submit_to_slide_time=4498) [2025-01-08 00:52:04.032743] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=95] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=11683, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:04.032958] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326323338688004, v:0}, new_max_version={val:1736326323338688004, v:0}, last_min_version={val:1736326323338688002, v:0}, last_max_version={val:1736326323338688002, v:0}, total_time_us=6251, delta_us=694267) [2025-01-08 00:52:04.033004] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=42] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326323338688004, v:0}, min_version={val:1736326323338688004, v:0}, max_version={val:1736326323338688004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:04.036895] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=154] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=17398579, this={dio_aligned_buf:{buf_write_offset:3453, buf_padding_size:643, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:71265, truncate_used_ts:17542}, log_block_size:67108864, io_fd:{first_id:-1, second_id:152, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=17395712, buf_len=586, total_write_size=33202, total_write_size_after_dio=405504, ob_pwrite_used_ts=2307595, count=91, each_pwrite_cost=25358) [2025-01-08 00:52:04.061809] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.075280] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=94] get number(ret=0, service_type_=0, range=1, base_id=1736326324071137000, start_id=1736326324071137000, end_id=1736326324071137001) [2025-01-08 00:52:04.077133] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58677][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4933-0-0] [lt=189] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:04.077257] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58677][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4933-0-0] [lt=120] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:04.077306] INFO [LIB] stat (utility.h:1210) [58677][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4933-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=23, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:04.077325] INFO [LIB] stat (utility.h:1210) [58677][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4933-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=51452, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:04.077333] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58677][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4933-0-0] [lt=6] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=3274, avg_log_submit_to_slide_time=51431) [2025-01-08 00:52:04.077556] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=976, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:04.077595] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18449, miss_cnt:0, cache_read_size:2256943, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:04.077678] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:04.077697] INFO [LIB] stat (utility.h:1210) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=18] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=8, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:04.078613] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=11683) [2025-01-08 00:52:04.083435] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=11] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1572800, replace_num=15728) [2025-01-08 00:52:04.083544] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=107] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:52:04.102656] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:04.136090] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:04.153887] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:202) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=7] query dest_info_ succ(dest_info=[], archive_dest_changed=false) [2025-01-08 00:52:04.153994] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=57] finish to resize log external storage handler(current_ts=1736326324153945, tenant_max_cpu=4, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:52:04.154012] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [50437][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=15] snapshot_traffic_info once [2025-01-08 00:52:04.157442] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:52:04.157677] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=336] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1}, {id:1003}, {id:1001}]) [2025-01-08 00:52:04.157750] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:04.157777] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-01-08 00:52:04.157787] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:04.157801] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:04.157824] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193828071392, flush_io_finished_ret_:0, flush_io_finished_round_:3810, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:04.158346] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:04.158375] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:04.162143] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.166445] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=68] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:04.166557] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=32] gc stale ls task succ [2025-01-08 00:52:04.170025] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB871A8-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:04.170282] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=136][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:04.170317] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:04.187379] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185456-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:04.187924] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:04.187957] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-01-08 00:52:04.187968] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:04.187982] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:04.187996] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193828096181, flush_io_finished_ret_:0, flush_io_finished_round_:1982, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:04.189789] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D0DA-0-0] [lt=34][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:04.210171] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BD5-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:04.210406] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BD5-0-0] [lt=218] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:04.212806] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:52:04.227007] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4677, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:04.227118] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=106][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2244535}], scn_array=[cnt:1, {val:1736326324007921000, v:0}], curr_size=122, accum_size=1220, time_cost=15024) [2025-01-08 00:52:04.227157] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84499837}], scn_array=[cnt:1, {val:1736326324034317001, v:0}], curr_size=122, accum_size=708, time_cost=11786) [2025-01-08 00:52:04.227394] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=272] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:04.227450] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=340, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:04.227809] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:04.236608] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AF98-0-0] [lt=16] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.794119787216186523e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=389, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.607656696478790082e+02) [2025-01-08 00:52:04.243041] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=58, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:04.243179] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:04.248167] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58679][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DB5-0-0] [lt=42] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3540) [2025-01-08 00:52:04.248414] INFO [LIB] stat (utility.h:1210) [58679][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DB5-0-0] [lt=204] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:04.248452] INFO [LIB] stat (utility.h:1210) [58679][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DB5-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=9881, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:04.253419] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58681][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:04.256903] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58682][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:04.257568] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:52:04.266916] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=176] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:52:04.267012] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.270412] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=105] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:19908, max_lsn:18746246, max_scn:{val:1736326323848021004, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:18746246}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:18746124}, last_submit_end_lsn:{lsn:18746246}, last_submit_log_id:19908, last_submit_log_pid:1, max_flushed_lsn:{lsn:18746124}, max_flushed_end_lsn:{lsn:18746246}, max_flushed_log_pid:1, committed_end_lsn:{lsn:18746246}, last_slide_log_id:19908, last_slide_scn:{val:1736326323848021004, v:0}, last_slide_lsn:{lsn:18746124}, last_slide_end_lsn:{lsn:18746246}, last_slide_log_pid:1, last_slide_log_accum_checksum:3323435329, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8fc454530}, [StateMgr]={this:0x2ad8fc458830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:19909, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324399554770, palf_role_change_cb:0x2ad8fc45f2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8fc455c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324399758968, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/meta", dir_fd:175, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8fc45aa38, flashback_version:0}, log_storage:{log_tail:{lsn:18746246}, readable_log_tail:{lsn:18746246}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48358522, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324399554279, this:0x2ad8fc458f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324399555093, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8fc4589b0}) [2025-01-08 00:52:04.278155] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7724] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326324207423, last_flushed_end_lsn:{lsn:18746246}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326324204126, last_flushed_end_lsn:{lsn:18746246}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326324218028, last_flushed_end_lsn:{lsn:18746246}}]) [2025-01-08 00:52:04.278335] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=161] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=19909, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326324207423, last_flushed_end_lsn:{lsn:18746246}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326324204126, last_flushed_end_lsn:{lsn:18746246}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326324218028, last_flushed_end_lsn:{lsn:18746246}}]) [2025-01-08 00:52:04.278430] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=61] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1002, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18544, max_lsn:2257187, max_scn:{val:1736326323859459002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2257187}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2257065}, last_submit_end_lsn:{lsn:2257187}, last_submit_log_id:18544, last_submit_log_pid:1, max_flushed_lsn:{lsn:2257065}, max_flushed_end_lsn:{lsn:2257187}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2257187}, last_slide_log_id:18544, last_slide_scn:{val:1736326323859459002, v:0}, last_slide_lsn:{lsn:2257065}, last_slide_end_lsn:{lsn:2257187}, last_slide_log_pid:1, last_slide_log_accum_checksum:1258508369, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad92435e530}, [StateMgr]={this:0x2ad924362830, self:"192.168.159.137:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.138:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18545, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad9243692f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326321459401, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326321459401, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad92435fc70}, [ModeMgr]={palf_id:1002, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436803442001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436803442000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/meta", dir_fd:184, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924364a38, flashback_version:0}, log_storage:{log_tail:{lsn:2257187}, readable_log_tail:{lsn:2257187}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64847581, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:-1, this:0x2ad924362f70}, [Reconfirm]={palf_id:1002, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9243629b0}) [2025-01-08 00:52:04.278921] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=481][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:52:04.278952] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF_DUMP](palf_id=1002, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:52:04.278972] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:52:04.278990] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] current log_task status(palf_id=1002, self="192.168.159.137:2882", log_id=18545, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:52:04.283551] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=22] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:04.283780] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:52:04.284918] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=27] ====== check clog disk timer task ====== [2025-01-08 00:52:04.284956] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=31] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:52:04.290149] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58683][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:04.295986] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=235][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84499959}], scn_array=[cnt:1, {val:1736326324241390000, v:0}], curr_size=586, accum_size=1294, time_cost=52290) [2025-01-08 00:52:04.299300] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43671911}], scn_array=[cnt:1, {val:1736326324227137000, v:0}], curr_size=589, accum_size=589, time_cost=54560) [2025-01-08 00:52:04.300992] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2257187}], scn_array=[cnt:1, {val:1736326323859459003, v:0}], curr_size=122, accum_size=366, time_cost=14626) [2025-01-08 00:52:04.302301] INFO [LIB] stat (utility.h:1210) [58646][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BD7-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:04.302328] INFO [LIB] stat (utility.h:1210) [58646][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BD7-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=47733, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:04.302337] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58646][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BD7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=116, avg_log_submit_to_flush_time=4798, avg_log_submit_to_slide_time=47617) [2025-01-08 00:52:04.302451] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:04.302484] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18351, miss_cnt:0, cache_read_size:2244657, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:04.303528] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6729068544, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:04.305860] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58680][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A972D-0-0] [lt=1866] init point thread id with(&point=0x5572d50e7740, idx_=1336, point=[thread id=58680, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58680) [2025-01-08 00:52:04.306520] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=36] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:52:04.317047] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread created(tenant_->id()=1, group_id_=1, token=6) [2025-01-08 00:52:04.319276] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.319318] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.335778] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBB5-0-0] [lt=56][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193829249561, cache_obj->added_lc()=false, cache_obj->get_object_id()=13706, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:04.336029] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=88] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:04.336589] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=60] PNIO [ratelimit] time: 1736326324336586, bytes: 0, bw: 0.000000 MB/s, add_ts: 1010375, add_bytes: 0 [2025-01-08 00:52:04.350975] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326324350353000, start_id=1736326324350353000, end_id=1736326324350353001) [2025-01-08 00:52:04.351056] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=61] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326324350962], gts:1736326324350353000, latest_srr:[mts=1736326324350962]}) [2025-01-08 00:52:04.351167] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=108] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:04.351402] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=83] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326324351252], gts:1736326324350353000, latest_srr:[mts=1736326324351252]}) [2025-01-08 00:52:04.351799] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326324351778], gts:1736326324350353000, latest_srr:[mts=1736326324351778]}) [2025-01-08 00:52:04.356033] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=16637952) [2025-01-08 00:52:04.361400] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8850, offset=2514712}, end_cursor:ObLogCursor{file_id=1, log_id=8851, offset=2515017}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d40050, len:4328, log_data_len:305, seq:8850, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:04.361952] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7C-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:04.362025] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7C-0-0] [lt=51] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326323848021005, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326323848021005, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:04.367044] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=122][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.373315] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8851, offset=2515017}, end_cursor:ObLogCursor{file_id=1, log_id=8852, offset=2515314}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d44050, len:4023, log_data_len:297, seq:8851, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:04.373867] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7C-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:04.373956] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7C-0-0] [lt=59] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326323848021005, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326323848021005, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:04.381696] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:04.382900] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:04.385444] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=38] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8852, offset=2515314}, end_cursor:ObLogCursor{file_id=1, log_id=8853, offset=2515611}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d48050, len:3726, log_data_len:297, seq:8852, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:04.386563] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8853, offset=2515611}, end_cursor:ObLogCursor{file_id=1, log_id=8854, offset=2515908}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d4c050, len:3429, log_data_len:297, seq:8853, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:04.398880] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1348, avg_group_log_size=122) [2025-01-08 00:52:04.398926] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1348) [2025-01-08 00:52:04.403807] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=99] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:04.407117] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=165] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:04.408336] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:04.408454] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44663, [delay/us]:prepare:19, schedule:905, submit:400, rt:7126, total:8452) [2025-01-08 00:52:04.408484] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:04.408869] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=268] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44650, [delay/us]: prepare:19, schedule:905, submit:400, device:7126, total:8452) [2025-01-08 00:52:04.408905] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:04.408952] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4864, iops:16, bw:77122, [delay/us]:prepare:10, schedule:3223, submit:129, rt:2242, total:5605) [2025-01-08 00:52:04.409037] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=78] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:04.409199] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=118] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4864, avg_iops:16, avg_bw:77112, [delay/us]: prepare:10, schedule:3223, submit:129, device:2242, total:5605) [2025-01-08 00:52:04.409275] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:04.409306] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20292, [delay/us]:prepare:7, schedule:22, submit:1, rt:836, total:867) [2025-01-08 00:52:04.409362] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4291, iops:42, bw:178573, [delay/us]:prepare:7, schedule:585, submit:247, rt:3129, total:3969) [2025-01-08 00:52:04.409721] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=350] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:04.409791] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=62] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20287, [delay/us]: prepare:7, schedule:22, submit:1, device:836, total:867) [2025-01-08 00:52:04.409830] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4291, avg_iops:42, avg_bw:178525, [delay/us]: prepare:7, schedule:585, submit:247, device:3129, total:3969) [2025-01-08 00:52:04.409862] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:04.410071] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=191] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4056, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:3072, total:3085) [2025-01-08 00:52:04.410389] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=285] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:04.410452] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=54] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4055, [delay/us]: prepare:0, schedule:0, submit:600000000, device:3072, total:3085) [2025-01-08 00:52:04.410481] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:04.410522] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:04.410538] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:04.410551] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:04.410561] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SERVER] net_in=322kB/s, net_out=240kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:04.411568] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8597, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:04.411709] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=130] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1149, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:04.411849] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:04.436718] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] no need to update min/max version(new_version={val:1736326324034317000, v:0}, min_version={val:1736326324034317000, v:0}, max_version={val:1736326324034317000, v:0}) [2025-01-08 00:52:04.457102] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:52:04.458581] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58618][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=43] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:52:04.460278] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=224] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.460336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.460387] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5332838253, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:04.460484] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=21] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:52:04.460499] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5332838253, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:04.468529] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.483939] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:52:04.484090] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/70:1121704 disk:308K/70:1121704 t1m2:40K/9:703545 t1m2g0:40K/9:703545 t1002m2:184K/43:488125 t1002m2g0:184K/43:488125 t1001m2:84K/18:2839637 t1001m2g0:84K/18:2839637 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:04.504315] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:04.505983] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3540) [2025-01-08 00:52:04.519435] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.536808] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] no need to update min/max version(new_version={val:1736326324034317000, v:0}, min_version={val:1736326324034317000, v:0}, max_version={val:1736326324034317000, v:0}) [2025-01-08 00:52:04.542549] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=92] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.547447] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.568633] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:52:04.568700] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.577536] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A493D-0-0] [lt=19][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18547, log_task={header:{begin_lsn:{lsn:2257431}, end_lsn:{lsn:2257553}, log_id:18547, min_scn:{val:1736326324358495001, v:0}, max_scn:{val:1736326324358495001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2257309}, prev_proposal_id:1, committed_end_lsn:{lsn:2257431}, data_checksum:1715074492, accum_checksum:2774655301, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326324476589, freeze_ts:1736326324476590, submit_ts:1736326324476604, flushed_ts:1736326324483772, gen_to_freeze cost time:1, gen_to_submit cost time:15, submit_to_flush cost time:7168}, fs_cb_begin_ts=1736326324577522, log_life_time=100933) [2025-01-08 00:52:04.581741] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=54] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:04.603775] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58680][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BDD-0-0] [lt=93][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18444, log_task={header:{begin_lsn:{lsn:2244901}, end_lsn:{lsn:2245023}, log_id:18444, min_scn:{val:1736326324343347001, v:0}, max_scn:{val:1736326324343347001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2244779}, prev_proposal_id:1, committed_end_lsn:{lsn:2244901}, data_checksum:940009244, accum_checksum:2944095019, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326324501320, freeze_ts:1736326324501320, submit_ts:1736326324501327, flushed_ts:1736326324506084, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:4757}, fs_cb_begin_ts=1736326324603746, log_life_time=102426) [2025-01-08 00:52:04.615779] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB625-0-0] [lt=38] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:04.615919] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB625-0-0] [lt=110] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326324343347002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326324343347002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:04.622794] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:112) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1A8BB625-0-0] [lt=74] adaptive update(Tenant ID=1001, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=387, Holding Memory(MB)=102, Tenant Remain Memory(MB)=474, Usable Remain Memory(MB)=284, Last Update Limit Timestamp=1736326324622095, Is Updated=true) [2025-01-08 00:52:04.629853] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB625-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326324629653000, start_id=1736326324629653000, end_id=1736326324629653001) [2025-01-08 00:52:04.634087] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=23] rpc statistics(total_trans_msg_count=4, total_batch_msg_count=0) [2025-01-08 00:52:04.639396] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=102] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326324343347000, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=296038, in_cluster_service=true, cluster_version={val:1736326323848021003, v:0}, min_cluster_version={val:1736326323848021003, v:0}, max_cluster_version={val:1736326323848021003, v:0}, get_cluster_version_err=0, cluster_version_delta=791364, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18763, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=552556912, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:52:04.645399] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58682][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DBA-0-0] [lt=63] init point thread id with(&point=0x5572d50e77c0, idx_=1338, point=[thread id=58682, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58682) [2025-01-08 00:52:04.659991] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47] worker thread began to shrink(tenant_->id()=1002, group_id_=19, token=0) [2025-01-08 00:52:04.661111] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58398][T1002_L0_G19][T1002][Y0-0000000000000000-0-0] [lt=15] worker thread exit(tenant_->id()=1002, workers_.get_size()=1) [2025-01-08 00:52:04.668913] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.671107] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CF-0-0] [lt=29] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:04.671152] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2CF-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326324358495002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326324358495002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:04.677474] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=50] PNIO [ratelimit] time: 1736326324677471, bytes: 886471892, bw: 0.320577 MB/s, add_ts: 1000241, add_bytes: 336230 [2025-01-08 00:52:04.686512] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=67] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:52:04.703412] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=66] current monitor number(seq_=-1) [2025-01-08 00:52:04.709789] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:04.769055] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.787868] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=78][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:52:04.788129] WDIAG read_multi_row (ob_table_access_helper.h:238) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=133][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:52:04.789797] INFO delete_row (ob_table_access_helper.h:339) [46331][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326324788149) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326324788149) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:52:04.789987] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=81] ====== tenant freeze timer task ====== [2025-01-08 00:52:04.836326] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=49] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:04.840648] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=74] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:52:04.843322] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58681][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DBC-0-0] [lt=279] init point thread id with(&point=0x5572d50e7780, idx_=1337, point=[thread id=58681, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58681) [2025-01-08 00:52:04.844349] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE97C-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326324843952000, start_id=1736326324843952000, end_id=1736326324843952001) [2025-01-08 00:52:04.845075] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=76] ====== checkpoint timer task ====== [2025-01-08 00:52:04.846667] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=34] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:1, active_tx_count:1, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:04.846826] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=158] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1736326324842588000, v:0}) [2025-01-08 00:52:04.846935] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=27] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339284252000, latest_log_ts:{val:1736326322666668001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:04.847015] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=76] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:52:04.847045] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:52:04.847058] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] get rec log scn(service_type_=0, rec_log_ts={val:1736326322666668001, v:0}) [2025-01-08 00:52:04.847065] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:52:04.847072] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:52:04.847084] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:04.847103] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:04.847116] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:52:04.847136] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:04.847156] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:52:04.847166] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:52:04.852313] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=18] ts response task statistics(alloc_count=61150, free_count=30574) [2025-01-08 00:52:04.852593] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=50] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=520, get_gts_with_stc_cnt=3, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=3, try_wait_gts_elapse_cnt=0) [2025-01-08 00:52:04.855037] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=25] gc stale ls task succ [2025-01-08 00:52:04.855256] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=114] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=654004224, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=419737600, wash_ratio=6, tenant_id=1001) [2025-01-08 00:52:04.855326] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=35] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:52:04.855346] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=17] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2899, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:52:04.855406] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:52:04.855505] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:52:04.855524] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:52:04.855571] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:04.855595] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:52:04.855604] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:04.855617] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:04.855630] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193828768722, flush_io_finished_ret_:0, flush_io_finished_round_:1987, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:04.856618] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:04.857725] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=62] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:52:04.857815] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=86] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:52:04.857854] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=36] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:52:04.857865] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:04.860430] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=61] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:04.860665] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:52:04.861624] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=956] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:04.867129] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=125] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:52:04.869255] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.872288] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=32] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:52:04.873968] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC443-0-0] [lt=33][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:04.876471] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE2-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:04.876526] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE2-0-0] [lt=51] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:04.882011] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:04.883424] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=68] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:04.885999] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:04.886679] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:52:04.897052] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.897094] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.897120] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.897269] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:04.899923] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58684][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:04.900047] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58684][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE4B-0-0] [lt=91] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:52:04.900071] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58684][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE4B-0-0] [lt=17] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:52:04.900122] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58684][T1002_L0_G19][T1002][YB42C0A89F8A-00062B2D1E69BE4B-0-0] [lt=48] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:52:04.900289] INFO [SERVER.OMT] recv_group_request (ob_tenant.cpp:1413) [46142][pnio1][T0][YB42C0A89F8A-00062B2D1E69BE4B-0-0] [lt=6] worker thread created(id()=1002, group->group_id_=19) [2025-01-08 00:52:04.904352] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=49] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:04.909782] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:04.910248] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:04.918975] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=47] ====== checkpoint timer task ====== [2025-01-08 00:52:04.919047] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=46] get_max_applied_scn(scn={val:1736326324839208000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84502205}, palf_committed_end_scn_:{val:1736326324839208000, v:0}, last_check_scn_:{val:1736326324839208000, v:0}, max_applied_cb_scn_:{val:1736326324839208000, v:0}}) [2025-01-08 00:52:04.919079] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84502205}, readable_log_tail:{lsn:84502205}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49707331, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:04.920340] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=261] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:04.920454] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=113] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:04.920586] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=53] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339254996000, latest_log_ts:{val:1736326322625057001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:04.920695] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=104] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339254996000, latest_log_ts:{val:1736326322625057001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:52:04.920784] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=81] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:52:04.920803] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=15] get rec log scn(service_type_=0, rec_log_ts={val:1736326322625057001, v:0}) [2025-01-08 00:52:04.920811] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:52:04.920818] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:52:04.920831] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:04.920849] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:04.920861] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:52:04.920884] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:04.920931] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=45] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:52:04.920945] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=12] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:52:04.963129] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=32] log compressor cycles once. (ret=0, cost_time=1234, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12794863616, fast_delete_log_mode=false) [2025-01-08 00:52:04.963378] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=52] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:52:04.964782] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C68552E-0-0] [lt=22] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=111) [2025-01-08 00:52:04.964977] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1736326324964167000, start_id=1736326324964167000, end_id=1736326324964167001) [2025-01-08 00:52:04.969481] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:04.971225] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=76] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:52:04.971251] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] try gc retain ctx [2025-01-08 00:52:04.971260] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:52:04.971284] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1736326324883584000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43681338}, palf_committed_end_scn_:{val:1736326324883584000, v:0}, last_check_scn_:{val:1736326324883584000, v:0}, max_applied_cb_scn_:{val:1736326324883584000, v:0}}) [2025-01-08 00:52:04.971311] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] finish update min start scn(max_decided_scn={val:1736326324883584000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326314453572000, v:0}, keep_alive_scn_:{val:1736326314657020001, v:0}, update_ts_:1736326324970366}) [2025-01-08 00:52:04.972036] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339284252000, latest_log_ts:{val:1736326322666668001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:04.972299] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=33] LogLoopThread round_cost_time(us)(round_cost_time=27) [2025-01-08 00:52:04.976781] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD96-0-0] [lt=26] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326325084904, keepalive_ts:-1}) [2025-01-08 00:52:04.976833] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD96-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326325084904, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:04.983078] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30] LogLoopThread round_cost_time(us)(round_cost_time=10) [2025-01-08 00:52:04.997339] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CCA-0-0] [lt=51][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:04.998654] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=39] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:52:04.999366] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CCA-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:05.000625] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:52:05.000889] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=44, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:05.000921] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:05.000939] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:52:05.000953] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:05.005017] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9743-0-0] [lt=55] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:52:05.006762] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:52:05.006790] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] try gc retain ctx [2025-01-08 00:52:05.006798] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:52:05.006840] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326324839208000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326314648302000, v:0}, keep_alive_scn_:{val:1736326314648302000, v:0}, update_ts_:1736326325005650}) [2025-01-08 00:52:05.007910] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339254996000, latest_log_ts:{val:1736326322625057001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:05.008166] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=86] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3067, avg_group_log_size=306) [2025-01-08 00:52:05.008511] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58640][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BE5-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:52:05.009559] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3067) [2025-01-08 00:52:05.011790] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58679][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DBD-0-0] [lt=16] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=15, avg_log_gen_to_submit_time=146, avg_log_submit_to_flush_time=11398, avg_log_submit_to_slide_time=13788) [2025-01-08 00:52:05.012030] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:52:05.012031] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3067, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:05.012705] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58670][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=23] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:52:05.043012] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=86, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:05.043133] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=10718, avg_group_log_size=630) [2025-01-08 00:52:05.043985] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=115] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326324839208000, v:0}, new_max_version={val:1736326324839208000, v:0}, last_min_version={val:1736326324350353002, v:0}, last_max_version={val:1736326324350353002, v:0}, total_time_us=6200, delta_us=204774) [2025-01-08 00:52:05.044082] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=93] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326324839208000, v:0}, min_version={val:1736326324839208000, v:0}, max_version={val:1736326324839208000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:05.044239] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=5352, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:05.044294] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=49] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=713, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:05.044636] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10717) [2025-01-08 00:52:05.044663] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=39, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:05.045666] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FAA-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=24, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:05.045702] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FAA-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=9450, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:05.045715] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FAA-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=1113, avg_log_gen_to_submit_time=1219, avg_log_submit_to_flush_time=6428, avg_log_submit_to_slide_time=8231) [2025-01-08 00:52:05.045806] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10718, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:05.059251] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=50] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:52:05.061257] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326325058459000, start_id=1736326325058459000, end_id=1736326325058459001) [2025-01-08 00:52:05.063525] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=42] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:52:05.065716] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326325065483000, start_id=1736326325065483000, end_id=1736326325065483001) [2025-01-08 00:52:05.065933] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=115] [APPLY STAT CB APPEND COST TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=153, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:52:05.065956] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=23] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=11236, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:52:05.065965] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:52:05.065972] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=-990, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:52:05.065979] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=10399, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:52:05.069512] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:05.070238] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BC5-0-0] [lt=25] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:05.077830] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4947-0-0] [lt=100] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:05.077932] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4947-0-0] [lt=90] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:05.078033] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4947-0-0] [lt=98] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:05.079365] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2884, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:05.079453] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:52:05.079480] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=156, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:05.079558] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=61, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:05.081256] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B0F-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:05.081280] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B0F-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15158, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:05.081291] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B0F-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=29, avg_log_submit_to_flush_time=3099, avg_log_submit_to_slide_time=15128) [2025-01-08 00:52:05.081313] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:05.081350] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18460, miss_cnt:0, cache_read_size:2258285, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:05.081532] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=39] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=61, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:05.081602] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=68] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:05.084372] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326324883584000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=200732, in_cluster_service=true, cluster_version={val:1736326324687841000, v:0}, min_cluster_version={val:1736326324687841000, v:0}, max_cluster_version={val:1736326324687841000, v:0}, get_cluster_version_err=0, cluster_version_delta=396475, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=19192, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736326314422615000, v:0}, local_cluster_delta=10661701, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:52:05.086880] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:52:05.088007] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=11264) [2025-01-08 00:52:05.094726] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:219) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:52:05.094800] INFO [COMMON] handle (memory_dump.cpp:538) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=72] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-01-08 00:52:05.096034] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] update_check_range(min_check_version=378, max_check_version=379, global_version=380) [2025-01-08 00:52:05.101571] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=81, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:05.101623] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:05.105855] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9746-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1765) [2025-01-08 00:52:05.106335] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=28] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5334097784, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:52:05.112532] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=16] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:05.113024] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=30] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:3, threads_idle_time:5334097785, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:52:05.114329] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2859, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:05.114458] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=127] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:52:05.114487] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=442, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:05.114550] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=81] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:05.114582] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6967, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:05.114605] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:05.114654] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:05.125600] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4443, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:05.125672] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=69][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18747645}], scn_array=[cnt:1, {val:1736326324964167001, v:0}], curr_size=122, accum_size=1765, time_cost=19817) [2025-01-08 00:52:05.125704] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1765) [2025-01-08 00:52:05.125724] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=515, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:05.125752] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2245624}], scn_array=[cnt:1, {val:1736326324843952002, v:0}], curr_size=122, accum_size=1089, time_cost=20401) [2025-01-08 00:52:05.125857] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:05.125895] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7152, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:05.125907] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=105, avg_log_submit_to_flush_time=5035, avg_log_submit_to_slide_time=7046) [2025-01-08 00:52:05.125932] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=26, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:05.126043] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1765, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:05.129561] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF1-0-0] [lt=29] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:52:05.129597] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF1-0-0] [lt=27] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=38) [2025-01-08 00:52:05.130805] INFO handle (memory_dump.cpp:665) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=33] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 154 261 0 1 2 68 1 0 1 4 1 1 0 1 5 43 5 0 1 7 1 2 0 1 8 48 1 0 1 12 7 3 0 1 13 1 1 0 1 14 1 2 0 1 15 1 1 0 1 18 4 3 0 1 20 1 2 0 500 0 94 238 0 500 7 7 4 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 2 2 0 500 17 9 7 0 500 22 79 62 0 500 23 23 10 0 508 0 2 10 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 101 230 0 1001 2 49 1 0 1001 4 1 1 0 1001 5 22 5 0 1001 8 30 1 0 1001 12 3 2 0 1001 13 1 1 0 1001 14 1 1 0 1001 18 4 3 0 1001 20 2 2 0 1002 0 163 245 0 1002 2 59 1 0 1002 4 2 1 0 1002 5 49 6 0 1002 7 11 2 0 1002 8 46 1 0 1002 12 2 3 0 1002 13 1 1 0 1002 14 1 1 0 1002 18 9 3 0 1002 20 1 2 0 cost_time: 34802 [2025-01-08 00:52:05.130866] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=36] [MEMORY] tenant: 1, limit: 2,147,483,648 hold: 1,049,079,808 cache_hold: 213,909,504 cache_used: 213,909,504 cache_item_count: 102 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 451,977,216 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 142,606,336 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 95,862,784 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 100,663,296 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 14,680,064 limit= 429,496,720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [2025-01-08 00:52:05.131271] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 451,977,216 used= 347,923,840 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,211,520 used= 31,196,160 count= 15 avg_used= 2,079,744 block_cnt= 15 chunk_cnt= 15 mod=CACHE_MAP_NODE [MEMORY] hold= 31,031,296 used= 30,968,256 count= 7 avg_used= 4,424,036 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 30,760,960 used= 30,611,008 count= 26 avg_used= 1,177,346 block_cnt= 26 chunk_cnt= 20 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,196,912 used= 16,035,017 count= 110 avg_used= 145,772 block_cnt= 37 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,268,096 used= 11,228,160 count= 4 avg_used= 2,807,040 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,618,560 used= 7,541,760 count= 10 avg_used= 754,176 block_cnt= 10 chunk_cnt= 8 mod=ResourceGroup [MEMORY] hold= 7,225,344 used= 6,209,536 count= 124 avg_used= 50,076 block_cnt= 124 chunk_cnt= 24 mod=[T]char [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 15 mod=CkptDgnMemCU [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,391,488 used= 2,971,968 count= 138 avg_used= 21,536 block_cnt= 138 chunk_cnt= 29 mod=DiagnosticInfo [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 2,990,080 used= 2,971,008 count= 1 avg_used= 2,971,008 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,780,352 used= 2,663,280 count= 586 avg_used= 4,544 block_cnt= 570 chunk_cnt= 29 mod=WorkerMap [MEMORY] hold= 2,768,896 used= 2,720,000 count= 13 avg_used= 209,230 block_cnt= 13 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,640 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 272 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,097,152 used= 2,095,616 count= 2 avg_used= 1,047,808 block_cnt= 2 chunk_cnt= 2 mod=TmpFileWBP [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,394,368 used= 1,372,072 count= 75 avg_used= 18,294 block_cnt= 75 chunk_cnt= 11 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,260,160 used= 1,199,264 count= 33 avg_used= 36,341 block_cnt= 33 chunk_cnt= 12 mod=CommSysVarFac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,181,184 count= 1 avg_used= 1,181,184 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 1,179,648 used= 1,175,808 count= 8 avg_used= 146,976 block_cnt= 8 chunk_cnt= 6 mod=SQLSessionInfo [MEMORY] hold= 960,192 used= 936,448 count= 102 avg_used= 9,180 block_cnt= 102 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 950,272 used= 466,784 count= 58 avg_used= 8,048 block_cnt= 58 chunk_cnt= 18 mod=[T]ObPerfEventR [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 5 mod=LogAlloc [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 516,224 used= 495,296 count= 109 avg_used= 4,544 block_cnt= 109 chunk_cnt= 26 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 393,280 count= 2 avg_used= 196,640 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 319,488 used= 315,800 count= 1 avg_used= 315,800 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 262,144 used= 261,120 count= 4 avg_used= 65,280 block_cnt= 4 chunk_cnt= 3 mod=MemTblObj [MEMORY] hold= 262,144 used= 150,272 count= 16 avg_used= 9,392 block_cnt= 16 chunk_cnt= 8 mod=[T]ObDSActionAr [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 212,992 used= 212,800 count= 1 avg_used= 212,800 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 203,840 used= 194,432 count= 49 avg_used= 3,968 block_cnt= 49 chunk_cnt= 9 mod=TinyAllocator [MEMORY] hold= 195,072 used= 190,464 count= 24 avg_used= 7,936 block_cnt= 24 chunk_cnt= 13 mod=LinearHashMapNo [MEMORY] hold= 178,816 used= 174,592 count= 22 avg_used= 7,936 block_cnt= 22 chunk_cnt= 10 mod=SqlSession [MEMORY] hold= 155,024 used= 2,177 count= 1,920 avg_used= 1 block_cnt= 19 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 154,432 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 13 mod=QueryAllocator [MEMORY] hold= 147,456 used= 131,072 count= 2 avg_used= 65,536 block_cnt= 2 chunk_cnt= 2 mod=MinorMergeMgr [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 98,304 used= 93,024 count= 1 avg_used= 93,024 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 94,336 used= 24,472 count= 347 avg_used= 70 block_cnt= 82 chunk_cnt= 27 mod=Coro [MEMORY] hold= 89,408 used= 87,296 count= 11 avg_used= 7,936 block_cnt= 11 chunk_cnt= 8 mod=SqlSessiVarMap [MEMORY] hold= 89,408 used= 87,296 count= 11 avg_used= 7,936 block_cnt= 11 chunk_cnt= 6 mod=PlanVaIdx [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 8 mod=TabletIDSetNode [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 64,000 used= 62,464 count= 8 avg_used= 7,808 block_cnt= 8 chunk_cnt= 5 mod=HashNodeLCSta [MEMORY] hold= 57,344 used= 50,944 count= 1 avg_used= 50,944 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 54,720 used= 51,840 count= 15 avg_used= 3,456 block_cnt= 12 chunk_cnt= 11 mod=CACHE_INST [MEMORY] hold= 53,600 used= 41,664 count= 62 avg_used= 672 block_cnt= 60 chunk_cnt= 20 mod=[T]ObWarningBuf [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=TltMemtablMgr [MEMORY] hold= 41,680 used= 29,120 count= 65 avg_used= 448 block_cnt= 34 chunk_cnt= 13 mod=OMT_Worker [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 38,496 used= 4,080 count= 170 avg_used= 24 block_cnt= 70 chunk_cnt= 28 mod=[T]MemoryContex [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,768 used= 31,744 count= 1 avg_used= 31,744 block_cnt= 1 chunk_cnt= 1 mod=ArrayIdxArr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PartCtx [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,640 used= 15,872 count= 4 avg_used= 3,968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=ColUsagNode [MEMORY] hold= 15,968 used= 15,584 count= 2 avg_used= 7,792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnMemNode [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 11,984 used= 11,592 count= 2 avg_used= 5,796 block_cnt= 2 chunk_cnt= 1 mod=StopWordSet [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 9,376 used= 8,984 count= 2 avg_used= 4,492 block_cnt= 2 chunk_cnt= 1 mod=FTPluginMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FTPluginHandle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,936 used= 5,808 count= 11 avg_used= 528 block_cnt= 11 chunk_cnt= 8 mod=SqlSessiQuerSql [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=HasNodGrpInfo [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,880 used= 4,664 count= 11 avg_used= 424 block_cnt= 10 chunk_cnt= 6 mod=PackStateMap [MEMORY] hold= 6,864 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,864 used= 4,664 count= 11 avg_used= 424 block_cnt= 11 chunk_cnt= 7 mod=ContextsMap [MEMORY] hold= 6,864 used= 4,664 count= 11 avg_used= 424 block_cnt= 11 chunk_cnt= 6 mod=SequenceMap [MEMORY] hold= 6,864 used= 4,664 count= 11 avg_used= 424 block_cnt= 11 chunk_cnt= 5 mod=SequenceIdMap [MEMORY] hold= 6,848 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 5 mod=DICollector [MEMORY] hold= 4,864 used= 4,288 count= 3 avg_used= 1,429 block_cnt= 2 chunk_cnt= 2 mod=NegParamIdex [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MvRefreshInfo [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucGrpInfo [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,152 used= 960 count= 1 avg_used= 960 block_cnt= 1 chunk_cnt= 1 mod=FTParserMgr [MEMORY] hold= 1,152 used= 768 count= 2 avg_used= 384 block_cnt= 2 chunk_cnt= 2 mod=TableLockNode [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 2 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 2 chunk_cnt= 2 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSIDDLVar [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 347,923,840 used= 340,304,746 count= 31,608 avg_used= 10,766 mod=SUMMARY [2025-01-08 00:52:05.131334] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=48] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 142,606,336 used= 141,492,224 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 block_cnt= 68 chunk_cnt= 68 mod=Memstore [MEMORY] hold= 141,492,224 used= 141,427,488 count= 68 avg_used= 2,079,816 mod=SUMMARY [2025-01-08 00:52:05.131355] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 729,088 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 729,088 used= 722,560 count= 1 avg_used= 722,560 mod=SUMMARY [2025-01-08 00:52:05.131376] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 95,862,784 used= 90,371,200 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83,884,032 used= 75,811,934 count= 2,908 avg_used= 26,070 block_cnt= 2,908 chunk_cnt= 43 mod=SqlPhyPlan [MEMORY] hold= 4,709,056 used= 4,468,656 count= 313 avg_used= 14,276 block_cnt= 313 chunk_cnt= 41 mod=SqlPlanCache [MEMORY] hold= 1,353,472 used= 1,234,048 count= 622 avg_used= 1,984 block_cnt= 208 chunk_cnt= 39 mod=CreateContext [MEMORY] hold= 422,656 used= 412,672 count= 52 avg_used= 7,936 block_cnt= 52 chunk_cnt= 12 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 90,371,200 used= 81,928,910 count= 3,897 avg_used= 21,023 mod=SUMMARY [2025-01-08 00:52:05.131396] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,368 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,272 used= 58,751 count= 259 avg_used= 226 block_cnt= 21 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,368 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-01-08 00:52:05.131412] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 100,663,296 used= 98,574,336 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 98,574,336 used= 98,390,976 count= 191 avg_used= 515,136 block_cnt= 191 chunk_cnt= 48 mod=CoStack [MEMORY] hold= 98,574,336 used= 98,390,976 count= 191 avg_used= 515,136 mod=SUMMARY [2025-01-08 00:52:05.131433] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 213,909,504 used= 213,909,504 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 213,909,504 used= 0 count= 102 avg_used= 0 mod=SUMMARY [2025-01-08 00:52:05.131450] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 14,680,064 used= 12,713,920 limit= 429,496,720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,036,352 used= 7,149,419 count= 109 avg_used= 65,591 block_cnt= 109 chunk_cnt= 5 mod=L_TabletPool [MEMORY] hold= 4,587,520 used= 4,569,600 count= 70 avg_used= 65,280 block_cnt= 70 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 12,713,920 used= 11,803,787 count= 181 avg_used= 65,214 mod=SUMMARY [2025-01-08 00:52:05.131463] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 2,097,152 used= 1,584,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 block_cnt= 195 chunk_cnt= 1 mod=MemtableCallbac [MEMORY] hold= 1,584,960 used= 1,547,520 count= 195 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:52:05.131476] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 16,256 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ExtInfoLog [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:52:05.131489] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 33,216 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 33,216 used= 28,849 count= 22 avg_used= 1,311 mod=SUMMARY [2025-01-08 00:52:05.131499] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 2,097,152 used= 0 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-01-08 00:52:05.131518] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 14,066,304 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 9,473,984 used= 9,445,184 count= 3 avg_used= 3,148,394 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 4,584,192 used= 4,475,904 count= 564 avg_used= 7,936 block_cnt= 564 chunk_cnt= 3 mod=TX_DATA_SLICE [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 14,066,304 used= 13,929,024 count= 568 avg_used= 24,522 mod=SUMMARY [2025-01-08 00:52:05.131564] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=29] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2,097,152 used= 73,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTable [MEMORY] hold= 73,152 used= 71,424 count= 9 avg_used= 7,936 mod=SUMMARY [2025-01-08 00:52:05.131604] INFO [LIB] operator() (ob_malloc_allocator.cpp:531) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 631,246,848 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 232,894,464 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 24,260,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 170,676,224 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98,557,952 limit= 9,223,372,036,854,775,807 [2025-01-08 00:52:05.131913] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:181) [46125][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 232,894,464 used= 185,992,176 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,397,376 used= 7,340,256 count= 7 avg_used= 1,048,608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7,259,504 used= 6,324,350 count= 1,032 avg_used= 6,128 block_cnt= 540 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,783,552 used= 5,752,960 count= 6 avg_used= 958,826 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 16 mod=Iteratorid()=1, group_id_=1, token=0) [2025-01-08 00:52:05.277903] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58683][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=45] worker thread exit(tenant_->id()=1, workers_.get_size()=6) [2025-01-08 00:52:05.283754] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=33] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:05.283380] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=99] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:33378, max_lsn:43683764, max_scn:{val:1736326325242522000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:43683764}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:1740724}}, last_submit_lsn:{lsn:43683175}, last_submit_end_lsn:{lsn:43683764}, last_submit_log_id:33378, last_submit_log_pid:1, max_flushed_lsn:{lsn:43683175}, max_flushed_end_lsn:{lsn:43683764}, max_flushed_log_pid:1, committed_end_lsn:{lsn:43683764}, last_slide_log_id:33378, last_slide_scn:{val:1736326325242522000, v:0}, last_slide_lsn:{lsn:43683175}, last_slide_end_lsn:{lsn:43683764}, last_slide_log_pid:1, last_slide_log_accum_checksum:1781162969, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad9083ee530}, [StateMgr]={this:0x2ad9083f2830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:33379, is_changing_config_with_arb:false, reconfirm_start_time_us:1736324348477669, palf_role_change_cb:0x2ad9083f92f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad9083efc70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736324348678844, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/meta", dir_fd:172, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad9083f4a38, flashback_version:0}, log_storage:{log_tail:{lsn:43683764}, readable_log_tail:{lsn:43683764}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23421004, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736324348477523, this:0x2ad9083f2f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.138:2882";"192.168.159.139:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.138:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736324348478003, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad9083f29b0}) [2025-01-08 00:52:05.283833] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=447] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326325245864, last_flushed_end_lsn:{lsn:43683764}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326325246949, last_flushed_end_lsn:{lsn:43683764}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326325247631, last_flushed_end_lsn:{lsn:43683764}}]) [2025-01-08 00:52:05.283861] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=24] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=33379, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326325245864, last_flushed_end_lsn:{lsn:43683764}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326325246949, last_flushed_end_lsn:{lsn:43683764}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326325247631, last_flushed_end_lsn:{lsn:43683764}}]) [2025-01-08 00:52:05.284165] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=67] ====== check clog disk timer task ====== [2025-01-08 00:52:05.284258] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=87] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:52:05.287048] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:52:05.287480] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853D2-0-0] [lt=30] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:52:05.287513] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50415][T1002_TimerWK0][T1002][YB42C0A89F89-00062B2D1D4853D2-0-0] [lt=25] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:52:05.302663] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BEB-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:05.302722] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BEB-0-0] [lt=37] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:05.302733] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BEB-0-0] [lt=11] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=43468, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:05.302742] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BEB-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=12, avg_log_submit_to_flush_time=4451, avg_log_submit_to_slide_time=43455) [2025-01-08 00:52:05.302911] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=95] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:05.302941] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18361, miss_cnt:0, cache_read_size:2245868, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:05.304488] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3837, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:05.304517] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1333) [2025-01-08 00:52:05.304535] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=311, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:05.304587] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=69, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:05.313431] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=102] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:05.313632] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=26] ====== check clog disk timer task ====== [2025-01-08 00:52:05.313660] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:52:05.319375] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.319421] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.336551] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:05.341131] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] Garbage Collector is running(seq_=383, gc_interval=10000000) [2025-01-08 00:52:05.341267] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=105] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:52:05.341321] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=43] gc_check_member_list_ cost time(ret=0, time_us=83) [2025-01-08 00:52:05.341338] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=12] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:52:05.341352] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:52:05.341365] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:52:05.341434] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=65] execute_gc cost time(ret=0, time_us=69) [2025-01-08 00:52:05.341443] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:52:05.341475] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=31] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-01-08 00:52:05.341497] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [46326][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-01-08 00:52:05.341494] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1736326325341467, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004881, add_bytes: 0 [2025-01-08 00:52:05.345575] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE97D-0-0] [lt=85] get number(ret=0, service_type_=0, range=1, base_id=1736326325344709000, start_id=1736326325344709000, end_id=1736326325344709001) [2025-01-08 00:52:05.357500] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:05.357788] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBB7-0-0] [lt=43][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193830271619, cache_obj->added_lc()=false, cache_obj->get_object_id()=13707, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:05.369786] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:05.382297] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:05.383741] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:05.402289] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1643, avg_group_log_size=164) [2025-01-08 00:52:05.402396] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1643) [2025-01-08 00:52:05.404929] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:172) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=17] Eliminate task evict sql audit(request_manager_->get_tenant_id()=1002, queue_size=524288, config_mem_limit_=96636764, request_manager_->get_size_used()=0, evict_high_size_level=471859, evict_low_size_level=419430, allocator->allocated()=70711296, evict_high_mem_level=75665245, evict_low_mem_level=54693724) [2025-01-08 00:52:05.405017] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=86] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:05.408595] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=97, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:05.408741] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:05.409926] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=27] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8854, offset=2515908}, end_cursor:ObLogCursor{file_id=1, log_id=8855, offset=2516213}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d50050, len:3132, log_data_len:305, seq:8854, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:05.410137] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:52:05.410740] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7D-0-0] [lt=99] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:05.410951] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7D-0-0] [lt=100] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326324964167003, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326324964167003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:05.411567] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1642, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:05.411638] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=78] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=36, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:05.411652] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=8604, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:05.413400] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:05.413448] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:12, bw:48906, [delay/us]:prepare:9, schedule:130, submit:136, rt:1532, total:1809) [2025-01-08 00:52:05.413463] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:05.413519] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:12, avg_bw:48920, [delay/us]: prepare:9, schedule:130, submit:136, device:1532, total:1809) [2025-01-08 00:52:05.413533] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:05.413572] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4864, iops:16, bw:77466, [delay/us]:prepare:7, schedule:83, submit:262, rt:1613, total:1967) [2025-01-08 00:52:05.413582] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:05.413618] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4864, avg_iops:16, avg_bw:77481, [delay/us]: prepare:7, schedule:83, submit:262, device:1613, total:1967) [2025-01-08 00:52:05.413629] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:05.413649] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:1, bw:4078, [delay/us]:prepare:5, schedule:22, submit:1, rt:2404, total:2432) [2025-01-08 00:52:05.413726] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:171287, [delay/us]:prepare:10, schedule:152, submit:470, rt:2070, total:2704) [2025-01-08 00:52:05.413740] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:05.413822] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=75] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:1, avg_bw:4079, [delay/us]: prepare:5, schedule:22, submit:1, device:2404, total:2432) [2025-01-08 00:52:05.413870] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:171339, [delay/us]: prepare:10, schedule:152, submit:470, device:2070, total:2704) [2025-01-08 00:52:05.413883] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:05.413904] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4080, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:500, total:516) [2025-01-08 00:52:05.413929] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:05.413959] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4081, [delay/us]: prepare:0, schedule:0, submit:600000000, device:500, total:516) [2025-01-08 00:52:05.414052] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:05.414080] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:05.414090] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:05.414099] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:05.414107] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=249kB/s, net_out=227kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:05.416758] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E7D-0-0] [lt=8] get number(ret=0, service_type_=0, range=1, base_id=1736326325412330000, start_id=1736326325412330000, end_id=1736326325412330001) [2025-01-08 00:52:05.416878] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58687][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:05.416930] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58687][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A5A-0-0] [lt=24] init point thread id with(&point=0x5572d50e7900, idx_=1343, point=[thread id=58687, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58687) [2025-01-08 00:52:05.417402] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1, group_id_=1, token=6) [2025-01-08 00:52:05.421818] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8855, offset=2516213}, end_cursor:ObLogCursor{file_id=1, log_id=8856, offset=2516510}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d54050, len:2827, log_data_len:297, seq:8855, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:05.422622] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7D-0-0] [lt=49] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:05.422669] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7D-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326324964167003, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326324964167003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:05.434143] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8856, offset=2516510}, end_cursor:ObLogCursor{file_id=1, log_id=8857, offset=2516807}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d58050, len:2530, log_data_len:297, seq:8856, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:05.435152] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8857, offset=2516807}, end_cursor:ObLogCursor{file_id=1, log_id=8858, offset=2517104}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d5c050, len:2233, log_data_len:297, seq:8857, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:05.439120] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1736326325240167000, v:0}, base_version={val:1736326325038765000, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326325240167000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326325439104, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326325240167000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326325443159, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326325240167000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1736326325503461, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:52:05.443718] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11264) [2025-01-08 00:52:05.445316] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=741, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:05.445409] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=38, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:05.459469] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.459543] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.459654] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:52:05.461772] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5335026978, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:05.461869] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:05.461886] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a1d10) [2025-01-08 00:52:05.461896] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5335026978, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:05.465781] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58672][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=79] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:52:05.473044] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:05.484374] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=35] ====== tenant freeze timer task ====== [2025-01-08 00:52:05.488073] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=37] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:308K/72:141093 disk:308K/72:141093 t1m2:52K/12:300295 t1m2g0:52K/12:300295 t1002m2:184K/44:127483 t1002m2g0:184K/44:127483 t1001m2:72K/16:59119 t1001m2g0:72K/16:59119 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:05.488110] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:52:05.491328] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:52:05.491372] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=38] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70495000, tenant_id=1001) [2025-01-08 00:52:05.491397] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:52:05.514039] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:05.519124] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=60] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.542556] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.544454] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.573188] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:05.573318] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326321459401, last_first_register_time_us=1736324449183670, register_time_us=1736326321459401, curr_time_us=1736326325573316) [2025-01-08 00:52:05.573414] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326325573350, last_submit_register_req_time_us=1736326325573350, register_parent_reason=1) [2025-01-08 00:52:05.573434] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:52:05.601987] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BC7-0-0] [lt=33] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:05.602444] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BC9-0-0] [lt=92] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:05.609285] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4197) [2025-01-08 00:52:05.644769] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB626-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:05.644826] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB626-0-0] [lt=39] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326325345853002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326325345853002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:05.658667] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=37] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:52:05.662976] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:52:05.666024] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=20] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:52:05.666063] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=27] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:52:05.666079] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=101) [2025-01-08 00:52:05.673272] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:05.677866] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185408-0-0] [lt=46] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:52:05.677923] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185408-0-0] [lt=47] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:52:05.679473] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=43] PNIO [ratelimit] time: 1736326325679470, bytes: 886677872, bw: 0.196046 MB/s, add_ts: 1001999, add_bytes: 205980 [2025-01-08 00:52:05.688323] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:52:05.689031] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=123] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:52:05.690723] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=125] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:05.697157] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [46340][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=112] XA scheduler heartbeat task statistics(avg_time=1) [2025-01-08 00:52:05.702576] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D0-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:05.702646] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D0-0-0] [lt=46] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326325358418003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326325358418003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:05.704136] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=131] current monitor number(seq_=-1) [2025-01-08 00:52:05.714683] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=94] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:05.721343] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D56-0-0] [lt=46] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:52:05.731781] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=47] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=43693439, this={dio_aligned_buf:{buf_write_offset:1923, buf_padding_size:2173, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:46840, truncate_used_ts:13666}, log_block_size:67108864, io_fd:{first_id:-1, second_id:160, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=43692032, buf_len=516, total_write_size=87483, total_write_size_after_dio=610304, ob_pwrite_used_ts=2889308, count=128, each_pwrite_cost=22572) [2025-01-08 00:52:05.762872] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:52:05.762998] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=112] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:52:05.763059] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=58] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:52:05.763066] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:52:05.763072] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=5] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:52:05.763079] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=96) [2025-01-08 00:52:05.767182] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=38] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:52:05.767324] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=134] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:52:05.767341] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=15] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:52:05.767352] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:05.770082] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=58] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:52:05.770986] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=895] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:05.773529] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:05.831983] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=22] blacklist_loop exec finished(cost_time=80, is_enabled=true, send_cnt=0) [2025-01-08 00:52:05.836888] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:05.853914] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:05.855211] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=93] gc stale ls task succ [2025-01-08 00:52:05.855810] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:05.855904] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=44] temporary file module use no blocks [2025-01-08 00:52:05.855917] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:05.855939] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:05.856050] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=108] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193829769501, flush_io_finished_ret_:0, flush_io_finished_round_:1988, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:05.857757] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:05.861428] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:05.873646] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=117][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:05.874603] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC446-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:05.878596] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE3-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:05.878644] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE3-0-0] [lt=45] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:05.879125] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46506][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BEDA-0-0] [lt=173] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:52:05.882728] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:05.884018] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:05.887816] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:05.888626] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:52:05.893450] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=48] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:05.893548] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=97] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:52:05.893691] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84997-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:05.893741] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84997-0-0] [lt=46] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:05.897274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.897313] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.897338] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.897362] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:05.900391] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=27] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:05.901782] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=52] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:05.901985] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=44] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326324430078000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:05.905627] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=119] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:05.910692] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=51] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:05.915315] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:05.916771] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=70] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1509936, clean_num=31457) [2025-01-08 00:52:05.957421] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=18] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:05.958831] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:05.958958] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=50] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326315839366000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:05.959015] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=55] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326315839366000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326315839366000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:52:05.973846] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:05.995154] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF75-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326326101619, keepalive_ts:-1}) [2025-01-08 00:52:05.995250] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF75-0-0] [lt=94] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326326101619, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:05.996588] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9] Garbage Collector is running(seq_=199, gc_interval=10000000) [2025-01-08 00:52:05.996643] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=29] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:52:05.996666] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=12] gc_check_member_list_ cost time(ret=0, time_us=47) [2025-01-08 00:52:05.996680] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:52:05.997657] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58679][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB100-0-0] [lt=23] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326326026364, keepalive_ts:-1}) [2025-01-08 00:52:05.997742] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58679][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB100-0-0] [lt=83] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326326026364, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:05.998813] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=6] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:52:05.998907] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=84] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:52:05.998922] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=13] execute_gc cost time(ret=0, time_us=17) [2025-01-08 00:52:05.998935] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50579][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyHandler start process [2025-01-08 00:52:06.003486] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF37-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326326031194, keepalive_ts:-1}) [2025-01-08 00:52:06.003582] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF37-0-0] [lt=93] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326326031194, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:06.007418] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:06.007452] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:06.007471] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:52:06.007485] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:06.009034] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D9-0-0] [lt=18] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326326035886, keepalive_ts:-1}) [2025-01-08 00:52:06.009065] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2D9-0-0] [lt=29] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326326035886, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:06.010004] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2DA-0-0] [lt=14] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326326036309, keepalive_ts:-1}) [2025-01-08 00:52:06.010042] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2DA-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326326036309, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:06.013008] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4084) [2025-01-08 00:52:06.013387] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CCB-0-0] [lt=105][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:06.014552] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B2C-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:06.014881] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A60-0-0] [lt=102] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=106, avg_log_submit_to_flush_time=1951, avg_log_submit_to_slide_time=4648) [2025-01-08 00:52:06.015871] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CCB-0-0] [lt=132][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:06.015936] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:52:06.016280] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4084, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:06.017790] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58677][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=26] worker thread exit(tenant_->id()=1002, workers_.get_size()=7) [2025-01-08 00:52:06.042483] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=4670, avg_group_log_size=389) [2025-01-08 00:52:06.045517] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=74, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:06.045596] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=59] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=10336, avg_group_log_size=646) [2025-01-08 00:52:06.046452] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326325842282000, v:0}, new_max_version={val:1736326325842282000, v:0}, last_min_version={val:1736326325440503001, v:0}, last_max_version={val:1736326325440503001, v:0}, total_time_us=6092, delta_us=204168) [2025-01-08 00:52:06.046487] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326325842282000, v:0}, min_version={val:1736326325842282000, v:0}, max_version={val:1736326325842282000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:06.047790] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=53] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=1530, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:06.047820] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=277, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:06.047910] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10336) [2025-01-08 00:52:06.047936] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=17, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:06.048920] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FBA-0-0] [lt=21] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=25, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:06.048940] INFO [LIB] stat (utility.h:1210) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FBA-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=3504, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:06.048949] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FBA-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=29, avg_log_gen_to_submit_time=116, avg_log_submit_to_flush_time=1870, avg_log_submit_to_slide_time=3388) [2025-01-08 00:52:06.049228] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10336, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:06.073845] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.080159] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A495B-0-0] [lt=26] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:06.080266] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A495B-0-0] [lt=93] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:06.080285] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A495B-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:06.081456] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1139, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:06.081528] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=70] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:06.081551] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=111, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:06.081634] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:06.082892] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB79-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:06.082971] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB79-0-0] [lt=77] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3147, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:06.082986] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DB79-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=28, avg_log_submit_to_flush_time=1299, avg_log_submit_to_slide_time=3119) [2025-01-08 00:52:06.083059] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:06.083107] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18470, miss_cnt:0, cache_read_size:2259505, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:06.083338] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=95, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:06.083362] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=9, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:06.088976] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=15] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:52:06.089809] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9789) [2025-01-08 00:52:06.104314] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BFB-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:52:06.107538] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=78, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:06.107574] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:06.109401] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1912) [2025-01-08 00:52:06.117318] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:06.136420] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:06.143642] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1736326325852717001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2225616}, palf_committed_end_scn_:{val:1736326325852717002, v:0}, last_check_scn_:{val:1736326325852717002, v:0}, max_applied_cb_scn_:{val:1736326325852717001, v:0}}) [2025-01-08 00:52:06.143696] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=50] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2225616}, readable_log_tail:{lsn:2225616}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64879152, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:06.157740] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:52:06.160469] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1003}, {id:1002}]) [2025-01-08 00:52:06.161372] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:06.162902] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=88] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:06.163031] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=53] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326325712447000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:06.163057] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326325712447000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326325712447000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:52:06.165719] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:06.165761] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:06.168550] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0E0-0-0] [lt=47][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:06.168857] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:06.168901] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=12] gc stale ls task succ [2025-01-08 00:52:06.171666] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB871B4-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:06.171718] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:06.171731] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:06.173979] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.192276] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=70] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:06.192320] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=36] temporary file module use no blocks [2025-01-08 00:52:06.192331] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:06.192345] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:06.192360] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193830105741, flush_io_finished_ret_:0, flush_io_finished_round_:1984, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:06.193884] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185458-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:06.208168] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=10] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:52:06.210123] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1701, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:06.210151] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2034) [2025-01-08 00:52:06.210169] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=160, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:06.210318] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:06.210653] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1883, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:06.210675] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:52:06.210689] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=126, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:06.210710] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:06.210987] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B31-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:06.211048] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B31-0-0] [lt=60] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3864, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:06.211120] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:06.213650] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B32-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:06.213672] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B32-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5091, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:06.213730] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B32-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=110, avg_log_submit_to_flush_time=1923, avg_log_submit_to_slide_time=4981) [2025-01-08 00:52:06.213813] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2034, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:06.225006] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=8] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:52:06.225688] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=669] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:52:06.225734] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=43] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:52:06.225833] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=95] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:06.228932] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=31] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:52:06.230947] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=41] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:52:06.232457] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=49] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:52:06.232498] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:52:06.233861] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084883-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:52:06.233954] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084883-0-0] [lt=46] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:52:06.233973] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084883-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:52:06.236982] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=12] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:52:06.238947] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=47] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:52:06.240306] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=53] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:52:06.241778] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=46] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:52:06.241800] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=21] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:52:06.243175] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084883-0-0] [lt=9] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:52:06.243291] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084883-0-0] [lt=18] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:52:06.243308] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084883-0-0] [lt=16] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:52:06.246670] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=12] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:52:06.248745] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=38] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:52:06.250084] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58643][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DCB-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2954) [2025-01-08 00:52:06.252339] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068AFB6-0-0] [lt=83] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.794119787216186523e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=389, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.607656696478790082e+02) [2025-01-08 00:52:06.255932] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=38] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:52:06.255972] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:52:06.258756] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084883-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:52:06.258963] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084883-0-0] [lt=156] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:52:06.258980] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084883-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:52:06.259001] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=11] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:52:06.259866] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=860] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:06.259944] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=70] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:52:06.259957] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=11] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:06.259980] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084883-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:52:06.259998] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084883-0-0] [lt=18] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:52:06.260009] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=6] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:52:06.260015] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:06.260032] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084883-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:52:06.260044] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084883-0-0] [lt=12] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:52:06.260053] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=6] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:52:06.260059] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:06.260074] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084883-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:52:06.260086] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084883-0-0] [lt=12] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:52:06.261423] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:52:06.261443] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=19] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:52:06.261453] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=9] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:52:06.261492] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:52:06.261505] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:52:06.261514] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:52:06.261526] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:52:06.261541] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084883-0-0] [lt=14] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:52:06.274152] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.283836] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=89] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1001, is_allowed_broadcast=true) [2025-01-08 00:52:06.284016] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=112] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:06.285373] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=31] ====== check clog disk timer task ====== [2025-01-08 00:52:06.285391] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=14] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:52:06.290927] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=36] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=0, replace_num=15728) [2025-01-08 00:52:06.290954] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:52:06.300577] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43693553}], scn_array=[cnt:1, {val:1736326326286131000, v:0}], curr_size=589, accum_size=1178, time_cost=11075) [2025-01-08 00:52:06.309719] INFO [LIB] stat (utility.h:1210) [58646][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BFF-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:06.309892] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58646][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3BFF-0-0] [lt=165] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:52:06.312524] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2076, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:06.312553] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:06.312629] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=145, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:06.312705] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=79, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:06.314502] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9769-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:06.314527] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9769-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4481, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:06.314537] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9769-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=34, avg_log_submit_to_flush_time=2235, avg_log_submit_to_slide_time=4446) [2025-01-08 00:52:06.314831] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:06.315034] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=136] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18372, miss_cnt:0, cache_read_size:2247210, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:06.316677] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=20] ls blacklist refresh finish(cost_time=25784) [2025-01-08 00:52:06.317916] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:06.320462] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.320617] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.337158] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:06.349025] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1736326326349021, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007554, add_bytes: 0 [2025-01-08 00:52:06.357894] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=17] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:06.379600] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBB9-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193831293431, cache_obj->added_lc()=false, cache_obj->get_object_id()=13708, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:06.379684] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.383995] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:06.384287] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=127] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:06.400165] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=78] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:52:06.405898] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:06.409120] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1912, avg_group_log_size=191) [2025-01-08 00:52:06.409169] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1912) [2025-01-08 00:52:06.411433] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=31] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:52:06.414338] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:06.414387] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:06.415338] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1407, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:06.417608] INFO [LIB] stat (utility.h:1210) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A64-0-0] [lt=23] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:06.417643] INFO [LIB] stat (utility.h:1210) [58643][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A64-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5252, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:06.418159] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:06.418387] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=137] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44834, [delay/us]:prepare:8, schedule:236, submit:51, rt:1056, total:1353) [2025-01-08 00:52:06.418424] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:06.418483] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44833, [delay/us]: prepare:8, schedule:236, submit:51, device:1056, total:1353) [2025-01-08 00:52:06.418497] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:06.418639] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:15, bw:69281, [delay/us]:prepare:7, schedule:62, submit:54, rt:1701, total:1825) [2025-01-08 00:52:06.418728] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:06.418945] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=52] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:15, avg_bw:69263, [delay/us]: prepare:7, schedule:62, submit:54, device:1701, total:1825) [2025-01-08 00:52:06.418982] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:06.419008] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12222, [delay/us]:prepare:6, schedule:14, submit:0, rt:775, total:796) [2025-01-08 00:52:06.419030] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:162966, [delay/us]:prepare:7, schedule:80, submit:134, rt:1422, total:1645) [2025-01-08 00:52:06.419058] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:06.419093] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:3, avg_bw:12223, [delay/us]: prepare:6, schedule:14, submit:0, device:775, total:796) [2025-01-08 00:52:06.419109] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:162985, [delay/us]: prepare:7, schedule:80, submit:134, device:1422, total:1645) [2025-01-08 00:52:06.419120] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:06.419139] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4074, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:526, total:535) [2025-01-08 00:52:06.419169] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:06.419287] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=111] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4074, [delay/us]: prepare:0, schedule:0, submit:600000000, device:526, total:535) [2025-01-08 00:52:06.419307] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:06.419323] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:06.419332] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:06.419340] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:06.419348] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=215kB/s, net_out=230kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:06.425158] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:52:06.428248] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58643][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1, workers_.get_size()=6) [2025-01-08 00:52:06.438106] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=42] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8858, offset=2517104}, end_cursor:ObLogCursor{file_id=1, log_id=8859, offset=2517409}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d64050, len:1936, log_data_len:305, seq:8858, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:06.438536] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7E-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:06.438581] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7E-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326325852717004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326325852717004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:06.445272] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1736326326444833000, start_id=1736326326444833000, end_id=1736326326444833001) [2025-01-08 00:52:06.446810] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=73] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:06.446808] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=131, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:06.452893] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=55] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8859, offset=2517409}, end_cursor:ObLogCursor{file_id=1, log_id=8860, offset=2517706}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d60050, len:1631, log_data_len:297, seq:8859, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:06.453559] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7E-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:06.453608] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7E-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326325852717004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326325852717004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:06.453856] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=67] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9040) [2025-01-08 00:52:06.458566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.459423] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.463921] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5337031495, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:06.464377] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=347] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:52:06.464415] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=37] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5337031495, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:06.466067] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:52:06.466461] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58622][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=48] worker thread exit(tenant_->id()=1001, workers_.get_size()=3) [2025-01-08 00:52:06.467098] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=41] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8860, offset=2517706}, end_cursor:ObLogCursor{file_id=1, log_id=8861, offset=2518003}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d6c050, len:1334, log_data_len:297, seq:8860, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:06.468049] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8861, offset=2518003}, end_cursor:ObLogCursor{file_id=1, log_id=8862, offset=2518300}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d70050, len:1037, log_data_len:297, seq:8861, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:06.480483] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.491122] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:52:06.496577] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBBB-0-0] [lt=33][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193831410409, cache_obj->added_lc()=false, cache_obj->get_object_id()=13709, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:06.505794] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46380][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBC0-0-0] [lt=16] get number(ret=0, service_type_=0, range=1, base_id=1736326326505137000, start_id=1736326326505137000, end_id=1736326326505137001) [2025-01-08 00:52:06.506437] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:296K/70:83629 disk:296K/70:83629 t1m2:48K/11:20761 t1m2g0:48K/11:20761 t1002m2:176K/44:110389 t1002m2g0:176K/44:110389 t1001m2:72K/15:51237 t1001m2g0:72K/15:51237 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:06.508455] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=38] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:52:06.518429] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:06.518558] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.541924] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.544423] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.577993] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326322462502, last_first_register_time_us=1736324454612308, register_time_us=1736326322462502, curr_time_us=1736326326577990) [2025-01-08 00:52:06.578053] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326326578023, last_submit_register_req_time_us=1736326326578023, register_parent_reason=1) [2025-01-08 00:52:06.578070] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:52:06.581555] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.620919] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBE1F-0-0] [lt=41][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193831534750, cache_obj->added_lc()=false, cache_obj->get_object_id()=13710, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:06.645744] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=5432) [2025-01-08 00:52:06.653519] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1736326326652417000, start_id=1736326326652417000, end_id=1736326326652417001) [2025-01-08 00:52:06.671710] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB627-0-0] [lt=75] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:06.671774] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB627-0-0] [lt=50] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326326346384002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326326346384002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:06.678335] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1003, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:18465, max_lsn:2247576, max_scn:{val:1736326326346384002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:2247576}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:2247454}, last_submit_end_lsn:{lsn:2247576}, last_submit_log_id:18465, last_submit_log_pid:1, max_flushed_lsn:{lsn:2247454}, max_flushed_end_lsn:{lsn:2247576}, max_flushed_log_pid:1, committed_end_lsn:{lsn:2247576}, last_slide_log_id:18465, last_slide_scn:{val:1736326326346384002, v:0}, last_slide_lsn:{lsn:2247454}, last_slide_end_lsn:{lsn:2247576}, last_slide_log_pid:1, last_slide_log_accum_checksum:3315963939, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad924fd6530}, [StateMgr]={this:0x2ad924fda830, self:"192.168.159.137:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.139:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:18466, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2ad924fe12f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1736326326578023, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1736326326578023, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad924fd7c70}, [ModeMgr]={palf_id:1003, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1736324436825152001, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1736324436825152000, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/meta", dir_fd:181, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad924fdca38, flashback_version:0}, log_storage:{log_tail:{lsn:2247576}, readable_log_tail:{lsn:2247576}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64857192, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, palf_epoch:4, last_purge_throttling_ts:-1, this:0x2ad924fdaf70}, [Reconfirm]={palf_id:1003, self:"192.168.159.137:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad924fda9b0}) [2025-01-08 00:52:06.678643] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=301][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:52:06.678661] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF_DUMP](palf_id=1003, self="192.168.159.137:2882", ack_info_list=[]) [2025-01-08 00:52:06.678673] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="192.168.159.137:2882", tmp_server={server:"192.168.159.137:2882", timestamp:1, flag:0}) [2025-01-08 00:52:06.678732] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=56] current log_task status(palf_id=1003, self="192.168.159.137:2882", log_id=18466, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-01-08 00:52:06.679553] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1736326326679550, bytes: 886906645, bw: 0.218157 MB/s, add_ts: 1000080, add_bytes: 228773 [2025-01-08 00:52:06.681701] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.687094] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] no need to update min/max version(new_version={val:1736326326251463000, v:0}, min_version={val:1736326326251463000, v:0}, max_version={val:1736326326251463000, v:0}) [2025-01-08 00:52:06.687440] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [46346][T1_ArcFetcher][T1][YB42C0A89F89-00062B2D1678458E-0-0] [lt=252] ObArchiveFetcher is running(thread_index=0) [2025-01-08 00:52:06.688385] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=24] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:52:06.691261] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:52:06.714939] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708517D-0-0] [lt=208] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:06.719049] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:06.748674] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D1-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:06.748893] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D1-0-0] [lt=201] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326326363406002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326326363406002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:06.781755] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.793996] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-01-08 00:52:06.798710] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1C3BBB59-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1736326326798489000, start_id=1736326326798489000, end_id=1736326326798489001) [2025-01-08 00:52:06.837459] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=92] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:06.843348] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=38] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:52:06.843375] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:52:06.843417] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=49] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=208) [2025-01-08 00:52:06.843423] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=40] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:52:06.843499] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=72] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=175) [2025-01-08 00:52:06.843873] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326326642579000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=201282, in_cluster_service=true, cluster_version={val:1736326326444833000, v:0}, min_cluster_version={val:1736326326444833000, v:0}, max_cluster_version={val:1736326326444833000, v:0}, get_cluster_version_err=0, cluster_version_delta=399028, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36954, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=581340848, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:52:06.849780] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=21] [APPLY STAT CB APPEND COST TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=155, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:52:06.849821] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=41] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=10045, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:52:06.849831] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:52:06.849839] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=-1308, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:52:06.849846] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=8891, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:52:06.855375] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=43] gc stale ls task succ [2025-01-08 00:52:06.855845] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=178] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=508, get_gts_with_stc_cnt=33, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=31, try_wait_gts_elapse_cnt=0) [2025-01-08 00:52:06.855891] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=44] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:06.855969] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=498, get_gts_with_stc_cnt=80, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=77, try_wait_gts_elapse_cnt=0) [2025-01-08 00:52:06.857657] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1D69BAA7-0-0] [lt=25] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326326857542, can_tenant_elr:true}) [2025-01-08 00:52:06.858330] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:06.862366] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:06.871514] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC449-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:06.880348] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE4-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:06.880405] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE4-0-0] [lt=53] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:06.881793] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=100][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.883433] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A496B-0-0] [lt=51][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18570, log_task={header:{begin_lsn:{lsn:2260228}, end_lsn:{lsn:2260350}, log_id:18570, min_scn:{val:1736326326363406004, v:0}, max_scn:{val:1736326326363406004, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2260115}, prev_proposal_id:1, committed_end_lsn:{lsn:2260228}, data_checksum:1715074492, accum_checksum:843525551, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326326783131, freeze_ts:1736326326783131, submit_ts:1736326326783142, flushed_ts:1736326326784907, gen_to_freeze cost time:0, gen_to_submit cost time:11, submit_to_flush cost time:1765}, fs_cb_begin_ts=1736326326883406, log_life_time=100275) [2025-01-08 00:52:06.884346] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=154] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:06.884485] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:06.887376] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:06.891468] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:52:06.897114] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.897221] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.897259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.897312] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:06.906280] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:06.911801] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:06.915577] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221588795, support_ssl=false) [2025-01-08 00:52:06.915752] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=49] sm conn init succ(conn.sessid_=3221588795, sess.client_addr_="192.168.159.136:39670") [2025-01-08 00:52:06.915797] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=37] accept one succ(*s={this:0x2ad8a0dfc030, session_id:3221588795, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:156}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:06.916763] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:06.916804] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:292) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=40] mysql login queue(mysql_queue->queue_.size()=1) [2025-01-08 00:52:06.916867] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8B-0-0] [lt=91] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221588795, conn.client_addr_port_=39670, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221588795, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221588795, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D17584A8B-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46333, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523403055100631, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:102, fixup_index:-1, fixup_ash_buffer:{this:0x2ad96229eaf8, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:52:06.918378] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8B-0-0] [lt=82][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:52:06.918454] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8B-0-0] [lt=24] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221588795, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:06.919569] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=93] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:06.920798] INFO [SERVER] process (obmp_init_db.cpp:53) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84ABA-0-0] [lt=51] init db(db_name=oceanbase) [2025-01-08 00:52:06.935116] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=119] new mysql sessid created(conn.sessid_=3221588796, support_ssl=false) [2025-01-08 00:52:06.935372] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=34] sm conn init succ(conn.sessid_=3221588796, sess.client_addr_="192.168.159.136:39674") [2025-01-08 00:52:06.935396] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x2ad8bb804030, session_id:3221588796, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:183}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:06.936000] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=22] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:06.936078] INFO [SERVER] check_common_property (obmp_connect.cpp:2034) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8D-0-0] [lt=26] construct session id(conn.client_sessid_=999292932, conn.sessid_=3221588796, conn.client_addr_port_=39674, conn.client_create_time_=1736323265987074, conn.proxy_sessid_=12398558792525217795, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221588796, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221588796, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-00062B2D17784A8D-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:46334, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:523403107039241, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:82, fixup_index:-1, fixup_ash_buffer:{this:0x2ad8c3f69658, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-01-08 00:52:06.937824] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:262) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8D-0-0] [lt=192][errcode=-4019] size is overflow(length=13312, pos=4, need_size=17034, ret=-4019) [2025-01-08 00:52:06.937891] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8D-0-0] [lt=18] MySQL LOGIN(direct_client_ip="192.168.159.136", client_ip=192.168.159.136, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221588796, proxy_sessid=12398558792525217795, client_sessid=999292932, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:06.940305] INFO [SERVER] process (obmp_init_db.cpp:53) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484B9F-0-0] [lt=23] init db(db_name=oceanbase) [2025-01-08 00:52:06.944649] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA2-0-0] [lt=22] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12358, add_timestamp:1736326326944633}) [2025-01-08 00:52:06.944695] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA2-0-0] [lt=26] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12396, add_timestamp:1736326326944693}) [2025-01-08 00:52:06.948235] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA2-0-0] [lt=11][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:06.948505] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BB-0-0] [lt=17] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12358, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326326948482, sql_renew_time:1736326326948482, is_mark_fail:false}]) [2025-01-08 00:52:06.948540] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BB-0-0] [lt=34] success to process renew task(task={tenant_id:1, table_id:12358, add_timestamp:1736326326944633}, locations=[{table_id:12358, partition_id:12358, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326326948482, sql_renew_time:1736326326948482, is_mark_fail:false}]) [2025-01-08 00:52:06.948569] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BC-0-0] [lt=16] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12396, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326326948564, sql_renew_time:1736326326948564, is_mark_fail:false}]) [2025-01-08 00:52:06.948585] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BC-0-0] [lt=15] success to process renew task(task={tenant_id:1, table_id:12396, add_timestamp:1736326326944693}, locations=[{table_id:12396, partition_id:12396, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326326948564, sql_renew_time:1736326326948564, is_mark_fail:false}]) [2025-01-08 00:52:06.950107] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA3-0-0] [lt=19] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11001, add_timestamp:1736326326950095}) [2025-01-08 00:52:06.950312] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BD-0-0] [lt=13] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11001, vtable_type={tenant_id:1, type:2}) [2025-01-08 00:52:06.952654] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BD-0-0] [lt=23] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11001, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326326952629, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:06.952682] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BD-0-0] [lt=27] success to process renew task(task={tenant_id:1, table_id:11001, add_timestamp:1736326326950095}, locations=[{table_id:11001, partition_id:0, partition_cnt:1, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326326952629, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:06.956370] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA3-0-0] [lt=23][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:06.978894] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=117] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:06.981951] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:06.997494] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF76-0-0] [lt=47] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326327103512, keepalive_ts:-1}) [2025-01-08 00:52:06.997536] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF76-0-0] [lt=40] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326327103512, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:07.000783] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58684][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1B3-0-0] [lt=20] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:52:07.000843] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58684][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1B3-0-0] [lt=53] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:52:07.000872] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58684][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB1B3-0-0] [lt=27] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:52:07.011417] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=97] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:07.011466] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:52:07.011487] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1098) [2025-01-08 00:52:07.011501] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:07.015551] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1709CA11-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326327014434000, start_id=1736326327014434000, end_id=1736326327014434001) [2025-01-08 00:52:07.016098] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=976) [2025-01-08 00:52:07.026572] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:195) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CCC-0-0] [lt=58][errcode=-4018] table_operator_ get_source failed(ret=-4018) [2025-01-08 00:52:07.027950] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:52:07.027983] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58646][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=17] worker thread exit(tenant_->id()=1002, workers_.get_size()=6) [2025-01-08 00:52:07.028371] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84511590}], scn_array=[cnt:1, {val:1736326327014434000, v:0}], curr_size=545, accum_size=5733, time_cost=12572) [2025-01-08 00:52:07.028460] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=87] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5733) [2025-01-08 00:52:07.031748] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58679][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DD4-0-0] [lt=17] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=100, avg_log_submit_to_flush_time=3153, avg_log_submit_to_slide_time=4463) [2025-01-08 00:52:07.032071] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5733, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:07.033087] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCF-0-0] [lt=30] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326327138449, keepalive_ts:-1}) [2025-01-08 00:52:07.033118] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58571][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBCF-0-0] [lt=29] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326327138449, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:07.034360] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CCC-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:07.035855] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CCC-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:07.041061] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2247942}], scn_array=[cnt:1, {val:1736326326848068001, v:0}], curr_size=122, accum_size=854, time_cost=12535) [2025-01-08 00:52:07.046096] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=5733, avg_group_log_size=477) [2025-01-08 00:52:07.057914] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84512135}], scn_array=[cnt:1, {val:1736326327043406000, v:0}], curr_size=586, accum_size=586, time_cost=11590) [2025-01-08 00:52:07.057992] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18751078}], scn_array=[cnt:1, {val:1736326327043406000, v:0}], curr_size=545, accum_size=1822, time_cost=13389) [2025-01-08 00:52:07.059303] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=74, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:07.059408] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=9394, avg_group_log_size=626) [2025-01-08 00:52:07.064553] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1736326326845800000, v:0}, new_max_version={val:1736326326845800000, v:0}, last_min_version={val:1736326326642579000, v:0}, last_max_version={val:1736326326642579000, v:0}, total_time_us=20278, delta_us=218750) [2025-01-08 00:52:07.064590] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1736326326845800000, v:0}, min_version={val:1736326326845800000, v:0}, max_version={val:1736326326845800000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:07.067749] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58571][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FC9-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9394) [2025-01-08 00:52:07.083398] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.086779] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A496F-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:07.086818] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A496F-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:52:07.086898] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A496F-0-0] [lt=75] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:52:07.089765] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=3238, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:07.089794] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43701181}], scn_array=[cnt:1, {val:1736326327055657000, v:0}], curr_size=589, accum_size=8806, time_cost=20490) [2025-01-08 00:52:07.089825] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=929, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:07.089875] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=17, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:07.089892] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=6301, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:07.089900] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=101, avg_log_submit_to_flush_time=4206, avg_log_submit_to_slide_time=6200) [2025-01-08 00:52:07.089921] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=21, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:07.090009] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9394, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:07.090572] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1201, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:07.090596] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:52:07.090612] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=384, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:07.090700] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=129] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:07.090792] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:52:07.092309] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=111] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:52:07.092554] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=184] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=98) [2025-01-08 00:52:07.101255] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:52:07.104240] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=79] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84512721}, readable_log_tail:{lsn:84512721}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49696815, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:07.104395] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=152] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:52:07.104565] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:52:07.114973] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C0F-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:07.115131] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=33] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=80, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:07.115216] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=83] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:07.120094] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=56] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6726930432, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:07.129433] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [50576][T1001_RFLWorker][T1001][YB42C0A89F89-00062B2D1C78458E-0-0] [lt=83] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:52:07.134733] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF2-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:52:07.134766] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF2-0-0] [lt=26] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=35) [2025-01-08 00:52:07.139311] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=51] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:07.155501] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2248064}], scn_array=[cnt:1, {val:1736326326848068002, v:0}], curr_size=122, accum_size=976, time_cost=33080) [2025-01-08 00:52:07.157782] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2226592}], scn_array=[cnt:1, {val:1736326327043406000, v:0}], curr_size=122, accum_size=976, time_cost=35277) [2025-01-08 00:52:07.158006] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=15] gc stale ls task succ [2025-01-08 00:52:07.160853] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1001}, {id:1002}]) [2025-01-08 00:52:07.161017] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:07.161044] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-01-08 00:52:07.161053] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:07.161066] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:07.161079] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193831074376, flush_io_finished_ret_:0, flush_io_finished_round_:3813, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:07.163748] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] Garbage Collector is running(seq_=200, gc_interval=10000000) [2025-01-08 00:52:07.163794] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=26] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:52:07.163808] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:52:07.165030] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:52:07.166442] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info(lsn={lsn:2260594}, scn={val:1736326326863875002, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326326986343, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2260594}, next_to_submit_scn_:{val:1736326326863875002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2260472}, end_lsn:{lsn:2260594}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2260716}, readable_log_tail:{lsn:2260716}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64844052, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326326863875001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326326863875001, v:0}, data_checksum:3348837161, flag:12324}}, init_mode_version:0, accumulate_checksum:2231126027, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326326863875001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18479, miss_cnt:0, cache_read_size:2260594, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:07.170130] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:07.170150] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:07.170555] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BE-0-0] [lt=15] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12274, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:07.171321] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=10] [APPLY STAT CB APPEND COST TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=105, this=0x2ad934926280, extra_info="") [2025-01-08 00:52:07.171347] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=26] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=7903, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:52:07.171356] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:52:07.171363] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=-1443, this=0x2ad934926910, extra_info="") [2025-01-08 00:52:07.171370] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=25, stat_interval=5000000, avg (count/cost)=6565, this=0x2ad934926b40, extra_info="") [2025-01-08 00:52:07.171963] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:07.172008] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:52:07.175570] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BE-0-0] [lt=19] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12274, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326327175547, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326327175547, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326327175547, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:07.175614] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BE-0-0] [lt=43] success to process renew task(task={tenant_id:1, table_id:12274, add_timestamp:1736326327169733}, locations=[{table_id:12274, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326327175547, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326327175547, sql_renew_time:0, is_mark_fail:false}, {table_id:12274, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326327175547, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:07.177449] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=26] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:52:07.177477] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=22] gc_check_member_list_ cost time(ret=0, time_us=13703) [2025-01-08 00:52:07.177493] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=1) [2025-01-08 00:52:07.178843] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0E3-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:07.178878] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:52:07.180313] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=20] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:52:07.181763] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:52:07.182991] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:52:07.183014] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:52:07.183030] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=14] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:52:07.183041] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:52:07.183052] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:52:07.183063] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=49) [2025-01-08 00:52:07.183070] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:52:07.183608] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.184734] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4971-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:07.184758] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4971-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=41668, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:07.184769] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4971-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=1620, avg_log_submit_to_slide_time=41653) [2025-01-08 00:52:07.184840] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=172] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:07.185006] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=157] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18480, miss_cnt:0, cache_read_size:2260716, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:07.185712] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB871BA-0-0] [lt=8][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:07.185753] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:07.185765] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:07.186416] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=8928) [2025-01-08 00:52:07.196111] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D19185459-0-0] [lt=104][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:07.220240] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C11-0-0] [lt=11][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18470, log_task={header:{begin_lsn:{lsn:2248064}, end_lsn:{lsn:2248186}, log_id:18470, min_scn:{val:1736326326848068002, v:0}, max_scn:{val:1736326326848068002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2247942}, prev_proposal_id:1, committed_end_lsn:{lsn:2248064}, data_checksum:940009244, accum_checksum:1401288556, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326327114929, freeze_ts:1736326327114930, submit_ts:1736326327114943, flushed_ts:1736326327155579, gen_to_freeze cost time:1, gen_to_submit cost time:14, submit_to_flush cost time:40636}, fs_cb_begin_ts=1736326327220175, log_life_time=105246) [2025-01-08 00:52:07.222610] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=83, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:07.222660] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:07.224099] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=82] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=5790, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:07.224127] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:52:07.224152] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=120, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:07.224526] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=62, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:07.224567] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=3334, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:07.224582] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1944) [2025-01-08 00:52:07.224598] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=154, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:07.224624] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:52:07.224638] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:07.226345] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B4D-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:07.226368] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B4D-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=4912, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:07.226379] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B4D-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=120, avg_log_submit_to_flush_time=3529, avg_log_submit_to_slide_time=4792) [2025-01-08 00:52:07.226686] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9784-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=44, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:07.226748] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9784-0-0] [lt=61] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=11099, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:07.226846] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1944, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:07.227025] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=111] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:07.248387] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=5733) [2025-01-08 00:52:07.272817] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=106] get_max_applied_scn(scn={val:1736326327043406000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18751745}, palf_committed_end_scn_:{val:1736326327043406001, v:0}, last_check_scn_:{val:1736326327043406001, v:0}, max_applied_cb_scn_:{val:1736326327043406000, v:0}}) [2025-01-08 00:52:07.272882] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=58] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18751745}, readable_log_tail:{lsn:18751745}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48353023, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:07.273030] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=81] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:52:07.277936] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=37] ====== traversal_flush timer task ====== [2025-01-08 00:52:07.278035] INFO [MDS] flush (mds_table_mgr.cpp:195) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B9-0-0] [lt=26] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326327043406001, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326323848021005, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326340087570000, latest_log_ts:{val:1736326323734680001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:07.278271] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B9-0-0] [lt=201] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326322838758003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326323848021005, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:07.278427] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B9-0-0] [lt=62] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326322860272003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326323848021005, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:07.278572] INFO [MDS] flush (mds_table_mgr.cpp:172) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B9-0-0] [lt=64] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326322849540004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326323848021005, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:07.278636] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C5854B9-0-0] [lt=61] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:52:07.283648] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=106] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:52:07.283685] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.284245] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:07.284644] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=27] ====== check clog disk timer task ====== [2025-01-08 00:52:07.284669] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:52:07.286931] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=31] LogLoopThread round_cost_time(us)(round_cost_time=11) [2025-01-08 00:52:07.301395] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:52:07.313939] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=32] ====== check clog disk timer task ====== [2025-01-08 00:52:07.313984] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=37] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:52:07.315623] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C13-0-0] [lt=59] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:07.315678] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C13-0-0] [lt=43] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:07.315716] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C13-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=25, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:07.315727] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C13-0-0] [lt=9] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=45575, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:07.315734] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C13-0-0] [lt=6] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=10, avg_log_submit_to_flush_time=7427, avg_log_submit_to_slide_time=45565) [2025-01-08 00:52:07.315833] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=101] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:07.315867] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18381, miss_cnt:0, cache_read_size:2248308, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:07.317274] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5795, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:07.317328] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:07.317360] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1015, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:07.317469] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:07.319586] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=81] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.319625] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.320810] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:07.337697] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:07.357005] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C79C2BE-0-0] [lt=21] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.770119237899780273e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=388, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.701485315953176496e+02) [2025-01-08 00:52:07.357322] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C79C2BE-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1736326327353672000, start_id=1736326327353672000, end_id=1736326327353672001) [2025-01-08 00:52:07.357453] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=53] PNIO [ratelimit] time: 1736326327357448, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008427, add_bytes: 0 [2025-01-08 00:52:07.357687] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326327357659], gts:1736326327353672000, latest_srr:[mts=1736326327357659]}) [2025-01-08 00:52:07.357782] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=78] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326327357767], gts:1736326327353672001, latest_srr:[mts=1736326327357767]}) [2025-01-08 00:52:07.358016] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326327358005], gts:1736326327353672000, latest_srr:[mts=1736326327358005]}) [2025-01-08 00:52:07.359019] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:07.383726] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.384559] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:07.384764] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:07.406511] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=98] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:52:07.410176] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBC1-0-0] [lt=18][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193832324008, cache_obj->added_lc()=false, cache_obj->get_object_id()=13711, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:07.412040] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=53] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:52:07.423111] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1944, avg_group_log_size=216) [2025-01-08 00:52:07.423210] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=94] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1944) [2025-01-08 00:52:07.430514] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=68, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:07.430552] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:07.431575] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:07.431683] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4840, iops:11, bw:52549, [delay/us]:prepare:11, schedule:58, submit:52, rt:2982, total:3104) [2025-01-08 00:52:07.431703] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:07.431771] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4841, avg_iops:11, avg_bw:52549, [delay/us]: prepare:11, schedule:58, submit:52, device:2982, total:3104) [2025-01-08 00:52:07.431785] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:07.431825] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:17, bw:76810, [delay/us]:prepare:7, schedule:76, submit:52, rt:2392, total:2529) [2025-01-08 00:52:07.431836] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:07.431873] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:76830, [delay/us]: prepare:7, schedule:76, submit:52, device:2392, total:2529) [2025-01-08 00:52:07.431884] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:07.431905] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16175, [delay/us]:prepare:7, schedule:501, submit:0, rt:758, total:1268) [2025-01-08 00:52:07.431927] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4311, iops:38, bw:161753, [delay/us]:prepare:11, schedule:97, submit:112, rt:3535, total:3758) [2025-01-08 00:52:07.431936] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:07.431964] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16175, [delay/us]: prepare:7, schedule:501, submit:0, device:758, total:1268) [2025-01-08 00:52:07.431979] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4312, avg_iops:38, avg_bw:161758, [delay/us]: prepare:11, schedule:97, submit:112, device:3535, total:3758) [2025-01-08 00:52:07.431989] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:07.432008] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4043, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:258, total:267) [2025-01-08 00:52:07.432031] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:07.432058] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4044, [delay/us]: prepare:0, schedule:0, submit:600000000, device:258, total:267) [2025-01-08 00:52:07.432075] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:07.432089] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:07.432098] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:07.432106] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:07.432114] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=241kB/s, net_out=275kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:07.433719] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3170, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:07.433776] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:07.433793] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=5139, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:07.434521] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:52:07.437902] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58679][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=27] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-01-08 00:52:07.440431] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=26] ====== traversal_flush timer task ====== [2025-01-08 00:52:07.440592] INFO [MDS] flush (mds_table_mgr.cpp:195) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84F1D-0-0] [lt=39] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326327391179000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339284252000, latest_log_ts:{val:1736326322666668001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:07.440719] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84F1D-0-0] [lt=121] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:52:07.444754] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] no need to update min/max version(new_version={val:1736326327043406000, v:0}, min_version={val:1736326327043406000, v:0}, max_version={val:1736326327043406000, v:0}) [2025-01-08 00:52:07.454648] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=28] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:52:07.458535] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.458996] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.465706] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=77] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5338764442, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:52:07.465761] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:07.465777] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad908599e10) [2025-01-08 00:52:07.465786] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5338764442, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:52:07.466746] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=38] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326326848068003, v:0}, base_version={val:1736326326848068001, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326326848068004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326327466728, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326326848068003, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326327521457, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326326848068003, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326327444614, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:52:07.468654] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=8928) [2025-01-08 00:52:07.472563] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8862, offset=2518300}, end_cursor:ObLogCursor{file_id=1, log_id=8863, offset=2518605}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d74050, len:740, log_data_len:305, seq:8862, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:07.472827] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7F-0-0] [lt=29] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:07.472866] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7F-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326327353672000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326327353672000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:07.476684] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:52:07.483388] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58571][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=25] worker thread exit(tenant_->id()=1001, workers_.get_size()=2) [2025-01-08 00:52:07.483913] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.488668] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=46] ====== tenant freeze timer task ====== [2025-01-08 00:52:07.489937] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2265178, this={dio_aligned_buf:{buf_write_offset:212, buf_padding_size:3884, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:27266, truncate_used_ts:5549}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2265088, buf_len=122, total_write_size=10708, total_write_size_after_dio=372736, ob_pwrite_used_ts=847629, count=88, each_pwrite_cost=9632) [2025-01-08 00:52:07.498536] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46370][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBC3-0-0] [lt=38][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193832412369, cache_obj->added_lc()=false, cache_obj->get_object_id()=13712, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:07.499408] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:52:07.499434] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70914440, tenant_id=1001) [2025-01-08 00:52:07.499456] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:52:07.501044] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=46] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8863, offset=2518605}, end_cursor:ObLogCursor{file_id=1, log_id=8864, offset=2518902}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d68050, len:435, log_data_len:297, seq:8863, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:07.501569] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:52:07.502118] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7F-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:07.502185] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E7F-0-0] [lt=49] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326327353672000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326327353672000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:07.504533] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=105] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:52:07.504560] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=140, cache_node_num=140) [2025-01-08 00:52:07.504572] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=140, cache_node_num=140) [2025-01-08 00:52:07.504582] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:07.507986] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=140, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}, {obj_id:2083, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221102}, {obj_id:2084, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2085, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2086, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2087, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2088, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2089, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2090, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:52:07.508409] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=419] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:07.512887] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:284K/66:132891 disk:284K/66:132891 t1m2:44K/10:70786 t1m2g0:44K/10:70786 t1002m2:168K/40:146401 t1002m2g0:168K/40:146401 t1001m2:72K/16:137932 t1001m2g0:72K/16:137932 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:07.518735] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.519309] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=51] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8864, offset=2518902}, end_cursor:ObLogCursor{file_id=1, log_id=8865, offset=2519199}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d78050, len:4234, log_data_len:297, seq:8864, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:07.521675] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=51] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8865, offset=2519199}, end_cursor:ObLogCursor{file_id=1, log_id=8866, offset=2519496}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d7c050, len:3937, log_data_len:297, seq:8865, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:07.522339] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=84] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:07.534253] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dfc030, session_id:3221588795, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326326922144, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:07.534320] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=58] kill and revert session(conn.sessid_=3221588795, proxy_sessid=12398558792525217795, ret=0) [2025-01-08 00:52:07.534336] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] sock ref clean, do destroy(*s={this:0x2ad8a0dfc030, session_id:3221588795, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326326922144, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:07.534371] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221588795, proxy_sessid=12398558792525217795, conn.tenant_id_=1) [2025-01-08 00:52:07.534498] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=124] connection close(sessid=3221588795, proxy_sessid=12398558792525217795, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84ABC-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:17038, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:12966409, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:592, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:12966410, flag_.flags:42, ext_flags:0}, obp_mysql_header_{len_:567, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:12966411, flag_.flags:58, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:12966411, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:12966411, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:12966411, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:12966411, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:12966411, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:12966411, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:07.535137] INFO [SERVER] run (obmp_disconnect.cpp:83) [46493][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=19] free session successfully(sessid=3221588795) [2025-01-08 00:52:07.535175] INFO [SERVER] run (obmp_disconnect.cpp:90) [46493][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=37] mark session id unused(sessid=3221588795) [2025-01-08 00:52:07.535609] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=143, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:07.535768] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:07.541871] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.543784] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.584049] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:52:07.584099] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.585741] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58620][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4979-0-0] [lt=43][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18577, log_task={header:{begin_lsn:{lsn:2261082}, end_lsn:{lsn:2261204}, log_id:18577, min_scn:{val:1736326327384872000, v:0}, max_scn:{val:1736326327384872000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2260960}, prev_proposal_id:1, committed_end_lsn:{lsn:2261082}, data_checksum:1715074492, accum_checksum:2810172260, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326327484749, freeze_ts:1736326327484749, submit_ts:1736326327484755, flushed_ts:1736326327489991, gen_to_freeze cost time:0, gen_to_submit cost time:6, submit_to_flush cost time:5236}, fs_cb_begin_ts=1736326327585706, log_life_time=100957) [2025-01-08 00:52:07.628105] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58690][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:07.628623] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread created(tenant_->id()=1002, group_id_=1, token=6) [2025-01-08 00:52:07.646909] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3499) [2025-01-08 00:52:07.682947] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=38] PNIO [ratelimit] time: 1736326327682943, bytes: 887195669, bw: 0.274703 MB/s, add_ts: 1003393, add_bytes: 289024 [2025-01-08 00:52:07.684258] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.686393] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A497B-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=18578, log_task={header:{begin_lsn:{lsn:2261204}, end_lsn:{lsn:2261326}, log_id:18578, min_scn:{val:1736326327384872001, v:0}, max_scn:{val:1736326327384872001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2261082}, prev_proposal_id:1, committed_end_lsn:{lsn:2261204}, data_checksum:1715074492, accum_checksum:2792248261, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326327585679, freeze_ts:1736326327585679, submit_ts:1736326327585690, flushed_ts:1736326327589359, gen_to_freeze cost time:0, gen_to_submit cost time:11, submit_to_flush cost time:3669}, fs_cb_begin_ts=1736326327686384, log_life_time=100705) [2025-01-08 00:52:07.696018] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB628-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:07.696073] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB628-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326327349144002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326327349144002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:07.701817] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:52:07.704522] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=101] current monitor number(seq_=-1) [2025-01-08 00:52:07.711542] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=18] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:52:07.722963] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=170] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:07.729087] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D57-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:52:07.782465] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:52:07.784419] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=118][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.793315] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D2-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:07.793360] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D2-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326327384872003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326327384872003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:07.838229] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=51] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:07.840888] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=225] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:52:07.855520] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=35] gc stale ls task succ [2025-01-08 00:52:07.855868] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=37] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=649809920, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=423931904, wash_ratio=6, tenant_id=1001) [2025-01-08 00:52:07.855961] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=51] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:52:07.855983] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2904, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:52:07.856049] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:52:07.856108] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:52:07.856275] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=164] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:52:07.857313] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=83] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:07.857346] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-01-08 00:52:07.857356] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:07.857370] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:07.857426] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193831771015, flush_io_finished_ret_:0, flush_io_finished_round_:1990, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:07.858346] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12] ts response task statistics(alloc_count=61222, free_count=30610) [2025-01-08 00:52:07.858378] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:07.859418] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=54] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=16637952) [2025-01-08 00:52:07.862531] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:07.870724] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC44C-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:07.873441] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:52:07.873483] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=39] get_min_unreplayed_log_info(lsn={lsn:2249040}, scn={val:1736326327349144005, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326327818685, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2249040}, next_to_submit_scn_:{val:1736326327349144005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2248918}, end_lsn:{lsn:2249040}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2249040}, readable_log_tail:{lsn:2249040}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64855728, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326327349144004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326327349144004, v:0}, data_checksum:2124447694, flag:5464}}, init_mode_version:0, accumulate_checksum:330968500, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326327349144004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18387, miss_cnt:0, cache_read_size:2249040, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:07.880276] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=32] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:52:07.882445] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE5-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:07.882502] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE5-0-0] [lt=53] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:07.884569] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.884855] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:07.884982] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:07.894699] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=57] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:07.894813] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=112] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:52:07.894872] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84998-0-0] [lt=15] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:07.894943] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84998-0-0] [lt=68] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:07.897336] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=43] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.897435] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.897480] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.897506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:07.909741] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=32] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:52:07.909882] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:52:07.910473] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=84] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=70711296) [2025-01-08 00:52:07.912389] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:07.923463] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:07.925073] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=57] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1541393, clean_num=31457) [2025-01-08 00:52:07.925145] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:52:07.925166] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] try gc retain ctx [2025-01-08 00:52:07.925172] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] try to retry start_working [2025-01-08 00:52:07.925284] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=4] finish update min start scn(max_decided_scn={val:1736326327353672004, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326317629447001, v:0}, keep_alive_scn_:{val:1736326322838758000, v:0}, update_ts_:1736326327922791}) [2025-01-08 00:52:07.926250] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=102] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326325345853002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326340087570000, latest_log_ts:{val:1736326323734680001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:07.926547] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=168] finish update min start scn(max_decided_scn={val:1736326327353672004, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326317629447001, v:0}, keep_alive_scn_:{val:1736326322838758000, v:0}, update_ts_:1736326327926417}) [2025-01-08 00:52:07.927616] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=36] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326322838758003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326325345853002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:07.927821] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=80] finish update min start scn(max_decided_scn={val:1736326327876795000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326321344654003, v:0}, keep_alive_scn_:{val:1736326326363406003, v:0}, update_ts_:1736326327927656}) [2025-01-08 00:52:07.928816] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326322860272003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326325345853002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:07.928913] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=85] finish update min start scn(max_decided_scn={val:1736326327851004000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326319864970000, v:0}, keep_alive_scn_:{val:1736326324843952001, v:0}, update_ts_:1736326327928796}) [2025-01-08 00:52:07.929693] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326322849540004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326325345853002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:07.982773] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=50] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:07.984829] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:07.996756] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:08.026492] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:08.026544] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:52:08.026615] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=66] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:52:08.026630] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:08.028959] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:52:08.032297] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3075) [2025-01-08 00:52:08.033570] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58681][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DDE-0-0] [lt=105] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=77, avg_log_submit_to_flush_time=3356, avg_log_submit_to_slide_time=4301) [2025-01-08 00:52:08.033849] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3075, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:08.044011] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CCD-0-0] [lt=54][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:08.045843] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CCD-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:08.048036] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3075, avg_group_log_size=307) [2025-01-08 00:52:08.069646] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=78] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=95, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:08.069692] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=10060, avg_group_log_size=591) [2025-01-08 00:52:08.071120] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=7] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10060) [2025-01-08 00:52:08.072740] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326327349144004, v:0}, new_max_version={val:1736326327349144004, v:0}, last_min_version={val:1736326327349144002, v:0}, last_max_version={val:1736326327349144002, v:0}, total_time_us=4996, delta_us=723594) [2025-01-08 00:52:08.072773] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326327349144004, v:0}, min_version={val:1736326327349144004, v:0}, max_version={val:1736326327349144004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:08.082414] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BC7-0-0] [lt=270] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:08.082613] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BC9-0-0] [lt=45] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:08.084908] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.086651] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4983-0-0] [lt=49] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:08.086725] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4983-0-0] [lt=71] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:08.110296] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=85] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:52:08.116904] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C23-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:08.120681] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=51, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:08.120707] INFO [LIB] stat (utility.h:1210) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=25] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=4, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:08.125588] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6741651456, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:08.139466] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:08.158447] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:52:08.159420] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0E6-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:08.161284] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=126] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1003}, {id:1001}]) [2025-01-08 00:52:08.161404] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:08.161428] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-01-08 00:52:08.161438] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:08.161452] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:08.161465] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193832074949, flush_io_finished_ret_:0, flush_io_finished_round_:3814, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:08.169334] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=40] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:52:08.169367] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=92) [2025-01-08 00:52:08.169494] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:08.169536] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:08.172238] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:08.172283] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:52:08.173486] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB871C0-0-0] [lt=11][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:08.173561] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:08.173575] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:08.184660] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=1698, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:08.184717] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=465, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:08.184782] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=13, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:08.185035] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.185861] INFO [LIB] stat (utility.h:1210) [58666][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3C73-0-0] [lt=38] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=12, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:08.185897] INFO [LIB] stat (utility.h:1210) [58666][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3C73-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=3754, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:08.185910] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58666][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3C73-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=119, avg_log_submit_to_flush_time=2653, avg_log_submit_to_slide_time=3635) [2025-01-08 00:52:08.186163] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10182, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:08.187045] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4985-0-0] [lt=26] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:08.188329] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1606, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:08.188354] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:52:08.188372] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=559, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:08.188441] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=47, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:08.189503] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B6A-0-0] [lt=38] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:08.189527] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B6A-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21764, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:08.189536] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B6A-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=2244, avg_log_submit_to_slide_time=21751) [2025-01-08 00:52:08.189613] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:08.189641] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18491, miss_cnt:0, cache_read_size:2262058, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:08.192692] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=57] get number(ret=0, service_type_=0, range=1, base_id=1736326328190603000, start_id=1736326328190603000, end_id=1736326328190603001) [2025-01-08 00:52:08.193199] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=87] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:08.193269] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=62] temporary file module use no blocks [2025-01-08 00:52:08.193281] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:08.193293] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:08.193306] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193832106894, flush_io_finished_ret_:0, flush_io_finished_round_:1986, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:08.194265] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10649) [2025-01-08 00:52:08.198329] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918545A-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:08.212267] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=17] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):280, sequence:1}) [2025-01-08 00:52:08.222318] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326318214673, end_time=1736326328222316, dag_status={}) [2025-01-08 00:52:08.222347] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:08.222360] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:08.222368] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:08.222376] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:08.222383] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:08.222390] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:08.222397] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:08.222403] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:08.222410] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:08.222417] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222428] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222436] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222444] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222452] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222461] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222469] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222477] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222485] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222493] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222501] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222509] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222517] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222525] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222533] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222541] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222549] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222556] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222565] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222573] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222581] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222589] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222596] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222604] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222612] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222620] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222627] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222635] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222643] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222651] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222659] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222667] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222674] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222682] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222690] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222698] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222705] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222713] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222721] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222729] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222737] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222745] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222753] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222760] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222768] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222776] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222784] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222792] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222799] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222807] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222814] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222822] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222830] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222838] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222845] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222853] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222861] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222869] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222876] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222884] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222892] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222900] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222907] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222915] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222923] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222979] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=55] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222988] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.222997] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.223005] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.223012] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.223020] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.223028] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.223036] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:08.223045] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:52:08.223052] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:52:08.223059] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:52:08.223064] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:52:08.223070] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:52:08.223075] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:52:08.223080] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:52:08.223085] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:52:08.223090] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:52:08.223095] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:52:08.223101] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:52:08.226707] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=58, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:08.226756] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:08.228992] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1636, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:08.229077] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=84] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:52:08.229099] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=945, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:08.229136] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:52:08.229608] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=457] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=32, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:08.229620] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4396, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:08.229628] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=79, avg_log_submit_to_flush_time=2687, avg_log_submit_to_slide_time=4316) [2025-01-08 00:52:08.229703] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=105, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:08.229766] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:08.231143] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2417, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:08.231253] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=108] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:52:08.231347] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=932, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:08.232361] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=27, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:08.232395] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4766, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:08.232412] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=64, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:08.232472] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:08.235929] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D17185142-0-0] [lt=45] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:52:08.250317] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3620) [2025-01-08 00:52:08.262357] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857DA-0-0] [lt=107][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193833176188, cache_obj->added_lc()=false, cache_obj->get_object_id()=13713, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:08.263302] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857DB-0-0] [lt=106][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:08.265417] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857DD-0-0] [lt=24][errcode=-4201] hashtable not init [2025-01-08 00:52:08.271687] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=21] ====== traversal_flush timer task ====== [2025-01-08 00:52:08.271832] INFO [MDS] flush (mds_table_mgr.cpp:195) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857ED-0-0] [lt=35] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326328248263000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339254996000, latest_log_ts:{val:1736326322625057001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:08.271975] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F857ED-0-0] [lt=136] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:52:08.284537] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=17] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:08.285153] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.285535] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=42] ====== check clog disk timer task ====== [2025-01-08 00:52:08.285574] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=33] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:52:08.311214] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:52:08.317225] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C27-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:08.317265] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58675][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C27-0-0] [lt=30] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:08.318740] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1539, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:08.318767] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:08.318802] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=90, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:08.318999] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.319039] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.319076] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=98, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:08.320664] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBA6-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:08.320690] INFO [LIB] stat (utility.h:1210) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBA6-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3495, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:08.320701] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58675][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBA6-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=26, avg_log_submit_to_flush_time=1806, avg_log_submit_to_slide_time=3469) [2025-01-08 00:52:08.320722] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:08.320778] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18392, miss_cnt:0, cache_read_size:2249650, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:08.326115] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=84] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:08.326329] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] ====== checkpoint timer task ====== [2025-01-08 00:52:08.327157] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:08.327259] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=100] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:08.327334] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326325345853002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326340087570000, latest_log_ts:{val:1736326323734680001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:08.327411] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=74] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:52:08.327431] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:52:08.327444] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326323734680001, v:0}) [2025-01-08 00:52:08.327452] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:52:08.327466] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:08.327484] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:08.327497] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:52:08.327515] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:08.327534] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:52:08.328213] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:08.328245] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=31] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:08.328283] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=16] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326322838758003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326325345853002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:08.328341] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=56] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:52:08.328353] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:52:08.328365] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:52:08.328379] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:52:08.328394] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2228047}, scn={val:1736326327857786003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:52:08.328737] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=34] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:52:08.331617] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8866, offset=2519496}, end_cursor:ObLogCursor{file_id=1, log_id=8867, offset=2519793}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d84050, len:3640, log_data_len:297, seq:8866, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:08.331699] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326327857786003, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:52:08.331730] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:08.331778] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=47] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:52:08.332621] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:08.332701] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=79] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:08.332744] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326322860272003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326325345853002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:08.332860] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=113] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:52:08.332873] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:52:08.332888] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:52:08.332903] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:52:08.332917] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2262180}, scn={val:1736326327876795004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:52:08.333785] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=21] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:52:08.338213] INFO [PALF] pread (log_reader.cpp:130) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=2883) [2025-01-08 00:52:08.338413] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:08.339428] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8867, offset=2519793}, end_cursor:ObLogCursor{file_id=1, log_id=8868, offset=2520090}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d80050, len:3343, log_data_len:297, seq:8867, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:08.339672] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326327876795004, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:52:08.339707] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:08.339728] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:52:08.340490] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:08.340526] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=35] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:08.340569] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326322849540004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326325345853002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:08.340662] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=91] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:52:08.340676] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:52:08.340692] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:52:08.340763] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=67] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:52:08.340778] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2249650}, scn={val:1736326327851004004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:52:08.341790] INFO [PALF] pread (log_reader.cpp:130) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=806) [2025-01-08 00:52:08.342893] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8868, offset=2520090}, end_cursor:ObLogCursor{file_id=1, log_id=8869, offset=2520387}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d88050, len:3046, log_data_len:297, seq:8868, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:08.342957] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326327851004004, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:52:08.342984] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:08.343016] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=30] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:52:08.343027] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:52:08.351476] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE990-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326328350729000, start_id=1736326328350729000, end_id=1736326328350729001) [2025-01-08 00:52:08.358992] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] get number(ret=0, service_type_=0, range=1, base_id=1736326328358286000, start_id=1736326328358286000, end_id=1736326328358286001) [2025-01-08 00:52:08.359673] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=18717696) [2025-01-08 00:52:08.364294] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1736326328364291, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006843, add_bytes: 0 [2025-01-08 00:52:08.376016] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=46] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:52:08.376335] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=47] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:52:08.377604] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=104][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=631246848, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:52:08.377723] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=115] === Run print tenant memory usage task === [2025-01-08 00:52:08.377774] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=18] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,612,800 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 633,344,000 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,328,376,867 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 159,752,580 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,049,079,808 max_mem_memstore_can_get_now= 798,762,986 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,328,376,867 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 70,914,440 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 649,809,920 max_mem_memstore_can_get_now= 354,572,248 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,328,376,867 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 240,672,760 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,114,390,528 max_mem_memstore_can_get_now= 1,203,363,818 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:52:08.378114] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=9] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,653,120,000 total_hold= 3,852,468,224 used= 3,477,770,240 freelists_hold= 175,349,760 total_maps= 1,638 total_unmaps= 248 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,612,800 virtual_memory_used= 4,825,022,464 [CHUNK_MGR] 2 MB_CACHE: hold= 73,400,320 count= 35 pushes= 1,850,220 pops= 1,850,185 maps= 1,507 unmaps= 243 [CHUNK_MGR] 4 MB_CACHE: hold= 7,745,536 count= 3 pushes= 6,154 pops= 6,151 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,170 pops= 3,166 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:52:08.378142] INFO print (ob_malloc_time_monitor.cpp:37) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=12] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 8459, delta_count= 2, avg_cost_time= 4229 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-01-08 00:52:08.379599] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1829F627-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326328378293000, start_id=1736326328378293000, end_id=1736326328378293001) [2025-01-08 00:52:08.379669] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F625-0-0] [lt=80] get number(ret=0, service_type_=0, range=1, base_id=1736326328378293000, start_id=1736326328378293000, end_id=1736326328378293001) [2025-01-08 00:52:08.385084] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=41] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:08.385366] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.385535] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:08.393841] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AF45-0-0] [lt=43] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.818119192123413086e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.515257032944482489e+02) [2025-01-08 00:52:08.410722] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:08.412808] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:08.428687] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:52:08.428723] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:52:08.432298] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:08.432333] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:08.437339] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46503][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBC5-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193833351173, cache_obj->added_lc()=false, cache_obj->get_object_id()=13714, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:08.437534] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:08.437569] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3179, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:08.438262] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:52:08.438400] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:08.438444] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:52890, [delay/us]:prepare:13, schedule:196, submit:47, rt:1076, total:1333) [2025-01-08 00:52:08.438459] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:08.438540] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=65] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52889, [delay/us]: prepare:13, schedule:196, submit:47, device:1076, total:1333) [2025-01-08 00:52:08.438555] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:08.438595] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:81369, [delay/us]:prepare:9, schedule:213, submit:82, rt:1224, total:1529) [2025-01-08 00:52:08.438607] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:08.438840] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=72] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:81357, [delay/us]: prepare:9, schedule:213, submit:82, device:1224, total:1529) [2025-01-08 00:52:08.438857] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:08.439056] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:7, bw:32540, [delay/us]:prepare:5, schedule:193, submit:0, rt:1200, total:1399) [2025-01-08 00:52:08.439098] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:12288 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4201, iops:39, bw:162702, [delay/us]:prepare:14, schedule:266, submit:262, rt:1168, total:1712) [2025-01-08 00:52:08.439110] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12202, [delay/us]:prepare:1, schedule:535, submit:672, rt:1265, total:2474) [2025-01-08 00:52:08.439120] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31033536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:08.439152] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4681, avg_iops:7, avg_bw:32534, [delay/us]: prepare:5, schedule:193, submit:0, device:1200, total:1399) [2025-01-08 00:52:08.439168] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12200, [delay/us]: prepare:1, schedule:535, submit:672, device:1265, total:2474) [2025-01-08 00:52:08.439445] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4201, avg_iops:39, avg_bw:162669, [delay/us]: prepare:14, schedule:266, submit:262, device:1168, total:1712) [2025-01-08 00:52:08.439517] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:08.439546] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4065, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:334, total:342) [2025-01-08 00:52:08.439574] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:08.439586] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2228169}], scn_array=[cnt:1, {val:1736326328358286000, v:0}], curr_size=122, accum_size=244, time_cost=10219) [2025-01-08 00:52:08.439667] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=86] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4065, [delay/us]: prepare:0, schedule:0, submit:600000000, device:334, total:342) [2025-01-08 00:52:08.439687] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:08.439705] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:08.439713] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:08.439784] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:08.439796] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=249kB/s, net_out=221kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:08.446329] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58682][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=40] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-01-08 00:52:08.450934] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1534, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:08.458474] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.458686] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.466477] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5339845585, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:52:08.466511] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:08.466523] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a0370) [2025-01-08 00:52:08.466532] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5339845585, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:52:08.472053] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10060) [2025-01-08 00:52:08.472710] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=93] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:52:08.485402] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=102][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.512097] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=29] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=15728, replace_num=15728) [2025-01-08 00:52:08.512131] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:52:08.518806] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:328K/76:198939 disk:328K/76:198939 t1m2:52K/12:173086 t1m2g0:52K/12:173086 t1002m2:208K/49:218608 t1002m2g0:208K/49:218608 t1001m2:68K/15:155370 t1001m2g0:68K/15:155370 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:08.519544] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.527139] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:08.527361] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8869, offset=2520387}, end_cursor:ObLogCursor{file_id=1, log_id=8870, offset=2520692}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d8c050, len:2749, log_data_len:305, seq:8869, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:08.527563] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E80-0-0] [lt=69] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:08.527606] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E80-0-0] [lt=26] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326328358286000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326328358286000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:08.539982] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8870, offset=2520692}, end_cursor:ObLogCursor{file_id=1, log_id=8871, offset=2520989}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d90050, len:2444, log_data_len:297, seq:8870, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:08.540509] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E80-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:08.540613] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E80-0-0] [lt=87] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326328358286000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326328358286000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:08.542310] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.544313] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.549224] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=76] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8871, offset=2520989}, end_cursor:ObLogCursor{file_id=1, log_id=8872, offset=2521286}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d94050, len:2147, log_data_len:297, seq:8871, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:08.550516] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8872, offset=2521286}, end_cursor:ObLogCursor{file_id=1, log_id=8873, offset=2521583}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d98050, len:1850, log_data_len:297, seq:8872, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:08.585537] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.629507] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BCB-0-0] [lt=17] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:08.632733] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:52:08.633274] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58675][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=12] worker thread exit(tenant_->id()=1002, workers_.get_size()=6) [2025-01-08 00:52:08.638988] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=99] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=90, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:08.639138] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=24, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:08.649991] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4084) [2025-01-08 00:52:08.685632] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.690550] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=54] PNIO [ratelimit] time: 1736326328690547, bytes: 887421912, bw: 0.214134 MB/s, add_ts: 1007604, add_bytes: 226243 [2025-01-08 00:52:08.701534] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=29] ====== tenant freeze timer task ====== [2025-01-08 00:52:08.704714] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=24] current monitor number(seq_=-1) [2025-01-08 00:52:08.716551] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=34] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:52:08.717503] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:52:08.725607] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB629-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:08.725726] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB629-0-0] [lt=31] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326328350729002, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326328350729002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:08.729754] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=115] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:08.752462] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB629-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326328751948000, start_id=1736326328751948000, end_id=1736326328751948001) [2025-01-08 00:52:08.752846] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:149) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB629-0-0] [lt=38] transaction rpc statistics(total_rt=4929, total_process=19, avg_rt=246) [2025-01-08 00:52:08.756170] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=22] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:52:08.775774] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] clear dtl interm result cost(us)(clear_cost=14, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:52:08.776719] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=110] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43718666}, readable_log_tail:{lsn:43718666}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23386102, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:08.776869] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=148] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:52:08.776904] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:52:08.785031] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58546][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB5D-0-0] [lt=53] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:52:08.785063] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58546][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB5D-0-0] [lt=25] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:52:08.786238] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.786422] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=131] clear dtl interm result cost(us)(clear_cost=4784, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:52:08.789911] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:62, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2051882560, recv_hp_rpc_cnt:5241, recv_np_rpc_cnt:104849, recv_lp_rpc_cnt:0, recv_mysql_cnt:9463, recv_task_cnt:3785, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10931701, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=117395 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1861, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:94881, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736326328438259}{group_id:10, queue_size:0, recv_req_cnt:11311, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2115, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326289381035}{group_id:2, queue_size:0, recv_req_cnt:30430, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11989, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:5020, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:27025, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=281 pcode=0x1502:cnt=204 pcode=0x717:cnt=179 pcode=0x150b:cnt=62 pcode=0x710:cnt=56, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:52:08.790685] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=772] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:15520588, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8277, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3617247, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=20, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:52:08.791444] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=758] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:15237233, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3603619, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:52:08.792304] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=858] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:18, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:895421550, recv_hp_rpc_cnt:3821, recv_np_rpc_cnt:38740, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:795093, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=66693 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:69193, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326327476682}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2236, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326279406832}{group_id:2, queue_size:0, recv_req_cnt:15821, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18958, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=298 pcode=0x51c:cnt=226 pcode=0x717:cnt=178 pcode=0x710:cnt=88 pcode=0x150b:cnt=60, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:52:08.793014] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=708] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:56, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9982978792, recv_hp_rpc_cnt:79, recv_np_rpc_cnt:35702, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3006920, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=25476 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:0, recv_req_cnt:7170, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326269316514}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:153519, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:5, nesting_worker_cnt:0, token_change:1736326328632731}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326302288855}{group_id:19, queue_size:0, recv_req_cnt:416, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326324899488}{group_id:2, queue_size:0, recv_req_cnt:37979, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:11343, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=366 pcode=0x1501:cnt=185 pcode=0x717:cnt=176 pcode=0x1516:cnt=172 pcode=0x150b:cnt=122, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:52:08.818822] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D3-0-0] [lt=34] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:08.818892] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58426][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D3-0-0] [lt=53] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326328378293004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326328378293004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:08.834639] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=132, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:52:08.834672] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=32] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=6089, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:52:08.834681] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:52:08.834688] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB EXECUTE TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=-1122, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:52:08.834696] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=5099, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:52:08.839324] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:08.847663] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=30] BlockGCTimerTask success(ret=0, cost_time_us=14, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:52:08.852685] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE994-0-0] [lt=48] get number(ret=0, service_type_=0, range=1, base_id=1736326328851931000, start_id=1736326328851931000, end_id=1736326328851931001) [2025-01-08 00:52:08.855882] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=49] gc stale ls task succ [2025-01-08 00:52:08.858360] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1736326328648832000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1736326328858324}, version_delta=209527) [2025-01-08 00:52:08.859784] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:08.860689] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=18717696) [2025-01-08 00:52:08.863763] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:08.875805] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC44F-0-0] [lt=52][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:08.878936] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58693][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:08.878990] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=668] worker thread created(tenant_->id()=1001, group_id_=1, token=2) [2025-01-08 00:52:08.882129] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BEDB-0-0] [lt=43] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:52:08.885090] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE6-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:08.885123] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE6-0-0] [lt=31] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:08.885946] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:08.885989] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=54] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:08.886377] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.897503] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=77] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.897543] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.897567] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.897590] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:08.908697] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=65] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:08.908872] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50915][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1609BCCA-0-0] [lt=50] get number(ret=0, service_type_=0, range=1, base_id=1736326328908788000, start_id=1736326328908788000, end_id=1736326328908788001) [2025-01-08 00:52:08.910415] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=53] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:08.910502] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=18] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326324430078000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:08.910528] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326324430078000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326324430078000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:52:08.910865] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=39] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:08.913151] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:52:08.926134] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=52] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:52:08.926253] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:52:08.930290] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6739554304, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:08.964896] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=28] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:08.966143] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=48] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:08.966317] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=100] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326326571484000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:08.978550] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=36][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:52:08.978653] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=19][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:52:08.978679] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=14][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:52:08.986122] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:08.986820] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=170][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:08.989741] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58693][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FE9-0-0] [lt=115] init point thread id with(&point=0x5572d50e7a80, idx_=1349, point=[thread id=58693, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58693) [2025-01-08 00:52:08.993215] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58681][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD97-0-0] [lt=131] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326329097364, keepalive_ts:-1}) [2025-01-08 00:52:08.993247] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58681][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD97-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326329097364, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:09.000922] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=44] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:43721304}, readable_log_tail:{lsn:43721304}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:23383464, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:09.030776] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:09.030812] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:09.030831] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:52:09.030844] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:09.032980] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:09.039701] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4085) [2025-01-08 00:52:09.039819] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=4, avg_log_gen_to_submit_time=77, avg_log_submit_to_flush_time=2313, avg_log_submit_to_slide_time=3475) [2025-01-08 00:52:09.039955] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4085, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:09.048593] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53] no need to update min/max version(new_version={val:1736326328648832000, v:0}, min_version={val:1736326328648832000, v:0}, max_version={val:1736326328648832000, v:0}) [2025-01-08 00:52:09.054737] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CCE-0-0] [lt=47][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:09.056673] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CCE-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:09.072601] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=68, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:09.072639] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=17, avg_log_batch_cnt=1, total_group_log_size=10063, avg_group_log_size=591) [2025-01-08 00:52:09.076087] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58694][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:09.076187] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58694][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3C83-0-0] [lt=67] init point thread id with(&point=0x5572d50e7c00, idx_=1355, point=[thread id=58694, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58694) [2025-01-08 00:52:09.076281] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FEB-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10063) [2025-01-08 00:52:09.076534] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread created(tenant_->id()=1001, group_id_=1, token=3) [2025-01-08 00:52:09.077771] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326328350729004, v:0}, new_max_version={val:1736326328350729004, v:0}, last_min_version={val:1736326328350729002, v:0}, last_max_version={val:1736326328350729002, v:0}, total_time_us=6946, delta_us=727040) [2025-01-08 00:52:09.077806] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326328350729004, v:0}, min_version={val:1736326328350729004, v:0}, max_version={val:1736326328350729004, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:09.086720] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.089123] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4995-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:52:09.089153] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4995-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:52:09.121787] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C37-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:09.126492] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:52:09.131480] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=87] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:09.136516] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=3621, avg_group_log_size=329) [2025-01-08 00:52:09.138624] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF3-0-0] [lt=26] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:52:09.138657] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF3-0-0] [lt=25] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=35) [2025-01-08 00:52:09.139668] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=41] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:09.158468] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=47] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:52:09.158769] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=15] gc stale ls task succ [2025-01-08 00:52:09.161553] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=78] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1002}, {id:1003}]) [2025-01-08 00:52:09.162931] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0E9-0-0] [lt=63][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:09.169110] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:09.169784] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:09.169804] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:09.170460] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:09.170550] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326325712447000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:09.172876] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:09.172918] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:52:09.176029] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB871C6-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:09.176107] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:09.176122] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:09.186897] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.188120] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT WRITE LOG COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=1965, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:09.188173] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=46] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=122, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:09.188473] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=45] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=43, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:09.189285] INFO [LIB] stat (utility.h:1210) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FEC-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=11, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:09.189307] INFO [LIB] stat (utility.h:1210) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FEC-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=3628, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:09.189318] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FEC-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=90, avg_log_submit_to_flush_time=2320, avg_log_submit_to_slide_time=3537) [2025-01-08 00:52:09.189518] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=209] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10063, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:09.189669] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4997-0-0] [lt=114] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:09.191534] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=1533, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:09.191559] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:52:09.191577] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=130, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:09.191627] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=61, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:09.193510] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B87-0-0] [lt=21] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=22, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:09.193533] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B87-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=14200, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:09.193542] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B87-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=1721, avg_log_submit_to_slide_time=14181) [2025-01-08 00:52:09.193659] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:09.193689] INFO [PALF] next (palf_iterator.h:183) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18500, miss_cnt:0, cache_read_size:2263156, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:09.193747] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=10] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=96, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:09.193763] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=15] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:09.200467] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918545B-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:09.204547] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58695][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:09.204611] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58695][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FED-0-0] [lt=38] init point thread id with(&point=0x5572d50e7c80, idx_=1357, point=[thread id=58695, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58695) [2025-01-08 00:52:09.204837] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10063) [2025-01-08 00:52:09.205299] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=93] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:09.205454] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1001, group_id_=1, token=4) [2025-01-08 00:52:09.223968] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326319214706, end_time=1736326329223966, dag_status={}) [2025-01-08 00:52:09.224003] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=32] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:09.224017] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:09.224025] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:09.224032] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:09.224040] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:09.224047] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:09.224054] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:09.224061] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:09.224067] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:52:09.224074] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224086] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224094] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224102] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224110] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224118] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224126] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224134] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224142] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224150] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224158] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224166] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224281] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=114] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224295] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224305] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224314] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224322] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224416] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224424] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224432] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224440] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224448] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224456] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224464] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224472] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224480] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224488] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224496] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224504] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224512] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224520] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224528] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224536] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224546] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224554] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224562] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224570] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224578] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224586] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224594] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224603] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224611] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224618] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224626] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224634] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224642] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224650] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224658] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224665] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224673] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224681] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224689] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224697] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224705] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224712] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224728] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224736] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224744] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224751] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224759] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224767] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224775] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224783] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224790] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224798] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224806] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224814] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224822] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224830] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224838] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224846] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:52:09.224855] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:52:09.224863] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:52:09.224870] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:52:09.224876] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:52:09.224881] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:52:09.224886] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:52:09.224891] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:52:09.224896] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:52:09.224902] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:52:09.224906] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:52:09.224913] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:52:09.230921] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=62, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:09.230972] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:09.232959] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3278, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:09.232987] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:09.233007] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=264, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:09.233051] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:09.233072] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:09.233789] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=79] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4196, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:09.233815] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:09.233832] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=109, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:09.233858] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=48, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:09.234438] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B88-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:09.234473] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B88-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5245, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:09.234598] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:09.235281] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B89-0-0] [lt=69] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:09.235305] INFO [LIB] stat (utility.h:1210) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B89-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5434, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:09.235316] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58620][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8B89-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=87, avg_log_submit_to_flush_time=3608, avg_log_submit_to_slide_time=5347) [2025-01-08 00:52:09.235872] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:09.267421] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x5572d527f900, last_check_time:1736326329267416, last_idle_time:15311669542, last_thread_count:4}, *pool={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15311669542, tenant_id:500}, qsize=-4, current_idle_time=15311669542) [2025-01-08 00:52:09.267478] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=56] [thread pool stat](pool_stat={pool:0x2ad8885abcf0, last_check_time:1736326329267477, last_idle_time:5340586663, last_thread_count:2}, *pool={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5340586663, tenant_id:500}, qsize=-1, current_idle_time=5340586663) [2025-01-08 00:52:09.267556] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=77] [thread pool stat](pool_stat={pool:0x2ad8885acaf0, last_check_time:1736326329267555, last_idle_time:3825393350, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885acaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3825393350, tenant_id:500}, qsize=-1, current_idle_time=3825393350) [2025-01-08 00:52:09.267566] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8881fc3b0, last_check_time:1736326329267566, last_idle_time:15245053091, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:15245053091, tenant_id:1}, qsize=-4, current_idle_time=15245053091) [2025-01-08 00:52:09.267576] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8885ad1f0, last_check_time:1736326329267576, last_idle_time:3737494401, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885ad1f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3737494401, tenant_id:500}, qsize=-1, current_idle_time=3737494401) [2025-01-08 00:52:09.267585] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8c0eb0e70, last_check_time:1736326329267585, last_idle_time:8040814812, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:8040814812, tenant_id:1002}, qsize=-4, current_idle_time=8040814812) [2025-01-08 00:52:09.267594] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] [thread pool stat](pool_stat={pool:0x2ad8f90b0e70, last_check_time:1736326329267594, last_idle_time:8003487143, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:8003487143, tenant_id:1001}, qsize=-4, current_idle_time=8003487143) [2025-01-08 00:52:09.284700] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=10] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:09.284942] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=17] ====== check clog disk timer task ====== [2025-01-08 00:52:09.284991] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=43] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:52:09.287272] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.292640] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A88-0-0] [lt=42] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326329292608}) [2025-01-08 00:52:09.292765] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BF-0-0] [lt=26] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:09.293113] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BF-0-0] [lt=25] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326329293090, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326329293090, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326329293090, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:09.293157] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859BF-0-0] [lt=44] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326329292608}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326329293090, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326329293090, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326329293090, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:09.315489] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=24] ====== check clog disk timer task ====== [2025-01-08 00:52:09.315526] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=31] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:52:09.315912] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=39] start to print blacklist info [2025-01-08 00:52:09.316146] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=21] ls blacklist refresh finish(cost_time=25084) [2025-01-08 00:52:09.318982] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.320319] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.320468] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C39-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:09.320477] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread began to shrink(tenant_->id()=1002, group_id_=18, token=0) [2025-01-08 00:52:09.320500] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C39-0-0] [lt=19] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:52:09.320558] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C39-0-0] [lt=15] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=11, avg_log_submit_to_flush_time=2853, avg_log_submit_to_slide_time=40211) [2025-01-08 00:52:09.320574] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C39-0-0] [lt=14][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18490, log_task={header:{begin_lsn:{lsn:2250504}, end_lsn:{lsn:2250626}, log_id:18490, min_scn:{val:1736326329113338000, v:0}, max_scn:{val:1736326329113338000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2250382}, prev_proposal_id:1, committed_end_lsn:{lsn:2250504}, data_checksum:940009244, accum_checksum:3662012681, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326329121763, freeze_ts:1736326329121763, submit_ts:1736326329121775, flushed_ts:1736326329125021, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:3246}, fs_cb_begin_ts=1736326329320523, log_life_time=198760) [2025-01-08 00:52:09.320652] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=976, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:09.320691] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18400, miss_cnt:0, cache_read_size:2250626, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:09.322043] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2313, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:09.322068] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:52:09.322102] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=103, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:09.322267] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=99, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:09.324991] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBB8-0-0] [lt=28] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:09.325018] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBB8-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=36252, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:09.326737] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:52:09.327232] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58426][T1002_L0_G18][T1002][Y0-0000000000000000-0-0] [lt=25] worker thread exit(tenant_->id()=1002, workers_.get_size()=1) [2025-01-08 00:52:09.332447] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:09.338457] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3540) [2025-01-08 00:52:09.339464] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=69] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:09.352781] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE995-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326329351725000, start_id=1736326329351725000, end_id=1736326329351725001) [2025-01-08 00:52:09.361359] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=80] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=18717696) [2025-01-08 00:52:09.371893] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=83] PNIO [ratelimit] time: 1736326329371888, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007597, add_bytes: 0 [2025-01-08 00:52:09.386236] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:09.386320] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:09.387470] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.398855] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AF54-0-0] [lt=32] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.818270540237426758e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=390, Throttle Trigger(MB)=234, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.514678795026297848e+02) [2025-01-08 00:52:09.403259] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7] ====== tenant freeze timer task ====== [2025-01-08 00:52:09.403288] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70704720, tenant_id=1001) [2025-01-08 00:52:09.403310] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:52:09.414620] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=124] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:09.417279] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:09.423704] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:52:09.423707] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58696][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:09.431269] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:09.431312] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:52:09.442277] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:09.442356] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:10, bw:40800, [delay/us]:prepare:6, schedule:103, submit:242, rt:1489, total:1841) [2025-01-08 00:52:09.442373] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:09.442450] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:10, avg_bw:40800, [delay/us]: prepare:6, schedule:103, submit:242, device:1489, total:1841) [2025-01-08 00:52:09.442466] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:09.442505] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, bw:73440, [delay/us]:prepare:10, schedule:308, submit:372, rt:1438, total:2129) [2025-01-08 00:52:09.442540] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:09.442583] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:73449, [delay/us]: prepare:10, schedule:308, submit:372, device:1438, total:2129) [2025-01-08 00:52:09.442595] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:09.442663] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=52] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16322, [delay/us]:prepare:5, schedule:18, submit:0, rt:1117, total:1141) [2025-01-08 00:52:09.442717] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4201, iops:39, bw:163224, [delay/us]:prepare:8, schedule:274, submit:884, rt:1348, total:2515) [2025-01-08 00:52:09.442729] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:09.442889] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=150] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16323, [delay/us]: prepare:5, schedule:18, submit:0, device:1117, total:1141) [2025-01-08 00:52:09.442911] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4201, avg_iops:39, avg_bw:163230, [delay/us]: prepare:8, schedule:274, submit:884, device:1348, total:2515) [2025-01-08 00:52:09.442923] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:09.443018] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=79] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4081, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:192, total:202) [2025-01-08 00:52:09.443048] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:09.443108] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4081, [delay/us]: prepare:0, schedule:0, submit:600000000, device:192, total:202) [2025-01-08 00:52:09.443129] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:09.443146] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:09.443155] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:09.443162] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:09.443170] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=327kB/s, net_out=254kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:09.444455] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:52:09.450386] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58674][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=86] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:52:09.458593] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.459366] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.459595] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBC7-0-0] [lt=21][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193834373430, cache_obj->added_lc()=false, cache_obj->get_object_id()=13715, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:09.487613] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.499585] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22] ====== tenant freeze timer task ====== [2025-01-08 00:52:09.519312] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.527019] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=91] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:52:09.532718] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=35] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:284K/69:242421 disk:284K/69:242421 t1m2:36K/9:106578 t1m2g0:36K/9:106578 t1002m2:176K/44:256062 t1002m2g0:176K/44:256062 t1001m2:72K/16:281320 t1001m2g0:72K/16:281320 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:09.533332] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=178] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:09.537000] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=63, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:09.537035] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:09.538526] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2071, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:09.539625] INFO [LIB] stat (utility.h:1210) [58681][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A86-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:09.539661] INFO [LIB] stat (utility.h:1210) [58681][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A86-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3342, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:09.542171] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.543983] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.549304] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] no need to update min/max version(new_version={val:1736326328859519001, v:0}, min_version={val:1736326328859519001, v:0}, max_version={val:1736326328859519001, v:0}) [2025-01-08 00:52:09.560739] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8873, offset=2521583}, end_cursor:ObLogCursor{file_id=1, log_id=8874, offset=2521888}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894d9c050, len:1553, log_data_len:305, seq:8873, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:09.561575] INFO [SERVER.OMT] recv_group_request (ob_tenant.cpp:1413) [46141][pnio1][T0][YB42C0A89F89-00062B2D1BC84E81-0-0] [lt=31] worker thread created(id()=1002, group->group_id_=18) [2025-01-08 00:52:09.561580] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58697][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:09.561635] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E81-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:09.561688] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E81-0-0] [lt=35] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326329359992000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326329359992000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:09.567055] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E81-0-0] [lt=22] get number(ret=0, service_type_=0, range=1, base_id=1736326329566670000, start_id=1736326329566670000, end_id=1736326329566670001) [2025-01-08 00:52:09.567301] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E81-0-0] [lt=63] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10105) [2025-01-08 00:52:09.571622] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8874, offset=2521888}, end_cursor:ObLogCursor{file_id=1, log_id=8875, offset=2522185}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894da0050, len:1248, log_data_len:297, seq:8874, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:09.572107] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E81-0-0] [lt=29] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:09.572165] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E81-0-0] [lt=45] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326329359992001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326329359992001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:09.579375] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8875, offset=2522185}, end_cursor:ObLogCursor{file_id=1, log_id=8876, offset=2522482}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894da4050, len:951, log_data_len:297, seq:8875, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:09.580501] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8876, offset=2522482}, end_cursor:ObLogCursor{file_id=1, log_id=8877, offset=2522779}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894da8050, len:654, log_data_len:297, seq:8876, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:09.587526] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326325573350, last_first_register_time_us=1736324449183670, register_time_us=1736326325573350, curr_time_us=1736326329587523) [2025-01-08 00:52:09.587660] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=35] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326329587563, last_submit_register_req_time_us=1736326329587563, register_parent_reason=1) [2025-01-08 00:52:09.587681] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:52:09.587788] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.624152] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58690][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A97C9-0-0] [lt=141] init point thread id with(&point=0x5572d50e79c0, idx_=1346, point=[thread id=58690, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58690) [2025-01-08 00:52:09.637398] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:52:09.638113] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58620][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=63] worker thread exit(tenant_->id()=1002, workers_.get_size()=5) [2025-01-08 00:52:09.651886] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=82] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3076) [2025-01-08 00:52:09.652813] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=166, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:09.652884] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=105] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=26, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:09.667922] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5340976587, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:52:09.668320] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=100] recycle one thread(this=0x2ad8885abcf0, total=2, remain=1) [2025-01-08 00:52:09.668365] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=43] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5340976587, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:52:09.671653] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326329353096001, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=318546, in_cluster_service=true, cluster_version={val:1736326328859519003, v:0}, min_cluster_version={val:1736326328859519003, v:0}, max_cluster_version={val:1736326328859519003, v:0}, get_cluster_version_err=0, cluster_version_delta=812123, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18813, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=557589169, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:52:09.687927] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.698160] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1736326329698156, bytes: 887741630, bw: 0.302604 MB/s, add_ts: 1007609, add_bytes: 319718 [2025-01-08 00:52:09.727248] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=46] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:52:09.733853] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=144] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:09.734510] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708517F-0-0] [lt=34] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:09.734771] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085181-0-0] [lt=25] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:09.736330] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D58-0-0] [lt=44] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:52:09.768464] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB62A-0-0] [lt=92] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:09.768516] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB62A-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326329353096003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326329353096003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:09.784979] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=113] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:52:09.788066] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.831998] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=24] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=1, inc_cnt=1, queue_size=1) [2025-01-08 00:52:09.841116] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=91] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:09.843279] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2229877}], scn_array=[cnt:1, {val:1736326329359992004, v:0}], curr_size=122, accum_size=732, time_cost=11465) [2025-01-08 00:52:09.844293] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18754786}], scn_array=[cnt:1, {val:1736326329359992004, v:0}], curr_size=122, accum_size=732, time_cost=12477) [2025-01-08 00:52:09.848391] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] ====== checkpoint timer task ====== [2025-01-08 00:52:09.849133] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=24] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:09.849177] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=43] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:09.849351] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=89] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339284252000, latest_log_ts:{val:1736326322666668001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:09.849432] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=77] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:52:09.849462] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:52:09.849476] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326322666668001, v:0}) [2025-01-08 00:52:09.849484] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:52:09.849492] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:52:09.849504] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:09.849521] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:09.849534] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:52:09.849553] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:09.849572] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:52:09.849582] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:52:09.856044] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:52:09.856875] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D4-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:09.856921] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D4-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326329380737004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326329380737004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:09.858744] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:09.858774] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-01-08 00:52:09.858784] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:09.858798] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:09.858900] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=100] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193833771832, flush_io_finished_ret_:0, flush_io_finished_round_:1992, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:09.861466] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=39] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:09.861878] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=491, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:52:09.863000] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=18717696) [2025-01-08 00:52:09.863228] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=64] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:52:09.863253] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:52:09.863264] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:52:09.863274] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:09.865929] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:52:09.866782] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=848] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:09.867086] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:09.867598] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=63] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:52:09.871839] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385727-0-0] [lt=12] MediumCheckTask(cost_ts=9) [2025-01-08 00:52:09.874466] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=45] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:52:09.883672] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC452-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:09.884781] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=27] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:52:09.886509] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:09.887577] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE7-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:09.887625] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE7-0-0] [lt=44] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:09.888246] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.888305] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:09.896282] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=49] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:09.896318] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=35] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:52:09.896466] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84999-0-0] [lt=14] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:09.896502] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A84999-0-0] [lt=32] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:09.897290] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.897974] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.898022] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.898047] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:09.899990] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:52:09.903550] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC5-0-0] [lt=25] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:52:09.903607] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58546][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEC5-0-0] [lt=48] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:52:09.914968] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:09.917423] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:52:09.921124] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=52] ====== checkpoint timer task ====== [2025-01-08 00:52:09.921228] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=36] get_max_applied_scn(scn={val:1736326329850793000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84522249}, palf_committed_end_scn_:{val:1736326329850793000, v:0}, last_check_scn_:{val:1736326329850793000, v:0}, max_applied_cb_scn_:{val:1736326329850793000, v:0}}) [2025-01-08 00:52:09.921287] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=53] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84522249}, readable_log_tail:{lsn:84522249}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49687287, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:09.922225] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=108] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:09.922283] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=56] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:52:09.922384] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=25] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339254996000, latest_log_ts:{val:1736326322625057001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:09.922489] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=99] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339254996000, latest_log_ts:{val:1736326322625057001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:52:09.922552] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=57] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:52:09.922590] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=34] get rec log scn(service_type_=0, rec_log_ts={val:1736326322625057001, v:0}) [2025-01-08 00:52:09.922602] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:52:09.922610] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:52:09.922624] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:09.922642] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:52:09.922655] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:52:09.922697] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:52:09.922720] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:52:09.922750] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=27] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:52:09.927549] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=85] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:52:09.933034] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18759004, this={dio_aligned_buf:{buf_write_offset:3542, buf_padding_size:554, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:33143, truncate_used_ts:6814}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=18755584, buf_len=122, total_write_size=15352, total_write_size_after_dio=417792, ob_pwrite_used_ts=407718, count=99, each_pwrite_cost=4118) [2025-01-08 00:52:09.933096] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2234095, this={dio_aligned_buf:{buf_write_offset:1897, buf_padding_size:2199, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:23176, truncate_used_ts:5251}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2232320, buf_len=122, total_write_size=11928, total_write_size_after_dio=413696, ob_pwrite_used_ts=437918, count=98, each_pwrite_cost=4468) [2025-01-08 00:52:09.934502] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=91] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:09.934586] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=53] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1572850, clean_num=31457) [2025-01-08 00:52:09.939670] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584AC0-0-0] [lt=52] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:09.940583] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB6B-0-0] [lt=21] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:52:09.940635] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB6B-0-0] [lt=43] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:52:09.963457] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=24] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12793626624, fast_delete_log_mode=false) [2025-01-08 00:52:09.963538] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAED3-0-0] [lt=28] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:52:09.963566] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58493][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAED3-0-0] [lt=22] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:52:09.968839] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=32] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:52:09.970489] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C68553C-0-0] [lt=27] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=106) [2025-01-08 00:52:09.970754] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1736326329969786000, start_id=1736326329969786000, end_id=1736326329969786001) [2025-01-08 00:52:09.987880] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=36] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:52:09.987934] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=53] try gc retain ctx [2025-01-08 00:52:09.987943] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-01-08 00:52:09.988038] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] get_max_applied_scn(scn={val:1736326329879463000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:43730111}, palf_committed_end_scn_:{val:1736326329879463000, v:0}, last_check_scn_:{val:1736326329879463000, v:0}, max_applied_cb_scn_:{val:1736326329879463000, v:0}}) [2025-01-08 00:52:09.988077] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=30] finish update min start scn(max_decided_scn={val:1736326329879463000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326314657020001, v:0}, keep_alive_scn_:{val:1736326324883584001, v:0}, update_ts_:1736326329987433}) [2025-01-08 00:52:09.989043] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339284252000, latest_log_ts:{val:1736326322666668001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:09.989510] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=215] [PALF STAT APPEND COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:09.989627] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=75] LogLoopThread round_cost_time(us)(round_cost_time=86) [2025-01-08 00:52:09.989706] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:09.992626] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58700][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:09.992723] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58700][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49A7-0-0] [lt=55] init point thread id with(&point=0x5572d50e7cc0, idx_=1358, point=[thread id=58700, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58700) [2025-01-08 00:52:09.993107] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42] worker thread created(tenant_->id()=1002, group_id_=1, token=5) [2025-01-08 00:52:09.998297] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=65] no need to update min/max version(new_version={val:1736326329598771000, v:0}, min_version={val:1736326329598771000, v:0}, max_version={val:1736326329598771000, v:0}) [2025-01-08 00:52:10.000946] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=42] LogLoopThread round_cost_time(us)(round_cost_time=10) [2025-01-08 00:52:10.001060] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=100] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:45224, max_lsn:84522249, max_scn:{val:1736326329850793000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84522249}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42579209}}, last_submit_lsn:{lsn:84521663}, last_submit_end_lsn:{lsn:84522249}, last_submit_log_id:45224, last_submit_log_pid:1, max_flushed_lsn:{lsn:84521663}, max_flushed_end_lsn:{lsn:84522249}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84522249}, last_slide_log_id:45224, last_slide_scn:{val:1736326329850793000, v:0}, last_slide_lsn:{lsn:84521663}, last_slide_end_lsn:{lsn:84522249}, last_slide_log_pid:1, last_slide_log_accum_checksum:21641629, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:45225, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84522249}, readable_log_tail:{lsn:84522249}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:49687287, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:52:10.001647] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=580] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326329857371, last_flushed_end_lsn:{lsn:84522249}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326329864050, last_flushed_end_lsn:{lsn:84522249}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326329859419, last_flushed_end_lsn:{lsn:84522249}}]) [2025-01-08 00:52:10.001672] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=45225, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326329857371, last_flushed_end_lsn:{lsn:84522249}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326329864050, last_flushed_end_lsn:{lsn:84522249}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326329859419, last_flushed_end_lsn:{lsn:84522249}}]) [2025-01-08 00:52:10.003041] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=10] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:52:10.008411] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58681][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB101-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326330035774, keepalive_ts:-1}) [2025-01-08 00:52:10.008465] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58681][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB101-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326330035774, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:10.010036] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF78-0-0] [lt=9] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326330113148, keepalive_ts:-1}) [2025-01-08 00:52:10.010087] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF78-0-0] [lt=49] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326330113148, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:10.012868] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF38-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326330040213, keepalive_ts:-1}) [2025-01-08 00:52:10.012920] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF38-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326330040213, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:10.024928] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2DC-0-0] [lt=16] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326330052573, keepalive_ts:-1}) [2025-01-08 00:52:10.024979] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2DC-0-0] [lt=50] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326330052573, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:10.025408] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2DD-0-0] [lt=67] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326330052670, keepalive_ts:-1}) [2025-01-08 00:52:10.025542] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2DD-0-0] [lt=132] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326330052670, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:10.032036] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=47, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:10.032073] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:10.032091] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:52:10.032106] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:10.035103] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A97D7-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:52:10.039946] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:52:10.039997] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=48] try gc retain ctx [2025-01-08 00:52:10.040008] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-01-08 00:52:10.040044] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326329850793000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326314648302000, v:0}, keep_alive_scn_:{val:1736326324850729000, v:0}, update_ts_:1736326330039695}) [2025-01-08 00:52:10.040737] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339254996000, latest_log_ts:{val:1736326322625057001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:52:10.042245] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3067) [2025-01-08 00:52:10.044681] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58701][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:10.044727] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58681][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A8B-0-0] [lt=19] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=14, avg_log_gen_to_submit_time=120, avg_log_submit_to_flush_time=2627, avg_log_submit_to_slide_time=3706) [2025-01-08 00:52:10.044764] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:52:10.044854] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3067, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:10.065986] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CCF-0-0] [lt=62][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:10.067093] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=14] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:52:10.068284] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CCF-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:10.068765] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326330068738000, start_id=1736326330068738000, end_id=1736326330068738001) [2025-01-08 00:52:10.069335] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=25] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:52:10.070968] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=64] get number(ret=0, service_type_=0, range=1, base_id=1736326330070918000, start_id=1736326330070918000, end_id=1736326330070918001) [2025-01-08 00:52:10.074244] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=71, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:10.074279] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9465, avg_group_log_size=591) [2025-01-08 00:52:10.074438] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=43] [APPLY STAT CB APPEND COST TIME](cur_stat_count=63, stat_interval=5000000, avg (count/cost)=127, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:52:10.074516] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=77] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=63, stat_interval=5000000, avg (count/cost)=4325, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:52:10.074528] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=63, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:52:10.074535] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB EXECUTE TIME](cur_stat_count=63, stat_interval=5000000, avg (count/cost)=-708, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:52:10.074542] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=63, stat_interval=5000000, avg (count/cost)=3744, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:52:10.077043] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9466) [2025-01-08 00:52:10.089836] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.092134] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49A9-0-0] [lt=45] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:10.092234] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49A9-0-0] [lt=96] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:10.098435] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326329879463000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=218955, in_cluster_service=true, cluster_version={val:1736326329598771000, v:0}, min_cluster_version={val:1736326329598771000, v:0}, max_cluster_version={val:1736326329598771000, v:0}, get_cluster_version_err=0, cluster_version_delta=499647, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=19241, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736326314422615000, v:0}, local_cluster_delta=15675803, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:52:10.104580] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=46] add async task(this=task_type:N9oceanbase10rootserver21ObPurgeRecyclebinTaskE, timeout_check:False) [2025-01-08 00:52:10.105853] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=126] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326329879463000, v:0}, new_max_version={val:1736326329879463000, v:0}, last_min_version={val:1736326329598771000, v:0}, last_max_version={val:1736326329598771000, v:0}, total_time_us=7268, delta_us=226386) [2025-01-08 00:52:10.105906] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=49] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326329879463000, v:0}, min_version={val:1736326329879463000, v:0}, max_version={val:1736326329879463000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:10.113270] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=18] success to update min reserved snapshot(reserved_snapshot=1736324525712447000, duration=1800, snapshot_gc_ts=1736326325712447000) [2025-01-08 00:52:10.116668] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=30] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324525712447000) [2025-01-08 00:52:10.119788] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] submit reserved snapshot log success(ls_id={id:1001}, new_snapshot_version=1736324525712447000) [2025-01-08 00:52:10.119815] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] update reserved snapshot finished(cost_ts=6550, reserved_snapshot=1736324525712447000) [2025-01-08 00:52:10.125465] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C49-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1089) [2025-01-08 00:52:10.127810] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:52:10.135341] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=73] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:10.139831] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=37] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:10.139898] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=7] finish to resize log external storage handler(current_ts=1736326330139894, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:52:10.139919] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=20] snapshot_traffic_info once [2025-01-08 00:52:10.158860] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:52:10.162705] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0EC-0-0] [lt=52][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:10.162962] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=67] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1002}, {id:1}]) [2025-01-08 00:52:10.163024] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:10.163065] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-01-08 00:52:10.163078] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:10.163091] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:10.163104] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193834075506, flush_io_finished_ret_:0, flush_io_finished_round_:3816, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:10.170309] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:10.170339] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:10.173105] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:10.173156] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:52:10.174016] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB871CC-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:10.174059] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:10.174071] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:10.185155] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A7A-0-0] [lt=48] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:10.189969] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.191954] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49AB-0-0] [lt=30] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:10.193434] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2258, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:10.193481] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:10.193505] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=120, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:10.193573] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=57, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:10.193653] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=36] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:10.193675] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-01-08 00:52:10.193685] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:10.193698] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:10.193711] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193834107328, flush_io_finished_ret_:0, flush_io_finished_round_:1988, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:10.195267] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BA7-0-0] [lt=45] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=27, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:10.195314] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BA7-0-0] [lt=46] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13508, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:10.195329] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BA7-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=15, avg_log_submit_to_flush_time=2470, avg_log_submit_to_slide_time=13493) [2025-01-08 00:52:10.195420] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:10.195453] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18510, miss_cnt:0, cache_read_size:2264376, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:10.195485] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] succ to submit log task to replay service(to_submit_lsn={lsn:2264310}, to_submit_scn={val:1736326329884917003, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326330195258, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2264376}, next_to_submit_scn_:{val:1736326329884917004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2264254}, end_lsn:{lsn:2264376}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2264376}, readable_log_tail:{lsn:2264376}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64840392, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326329884917003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326329884917003, v:0}, data_checksum:3348837161, flag:49387}}, init_mode_version:0, accumulate_checksum:563993193, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326329884917003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18510, miss_cnt:0, cache_read_size:2264376, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:10.195579] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=46, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:10.195590] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:10.203069] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918545C-0-0] [lt=222][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:10.221449] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A7B-0-0] [lt=111] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:10.223532] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=31] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:52:10.233868] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=81, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:10.233925] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:10.234882] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=77] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2294, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:10.234912] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:52:10.234932] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=62, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:10.234998] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:10.235295] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A7C-0-0] [lt=27] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:10.236769] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2587, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:10.237093] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=322] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1755) [2025-01-08 00:52:10.237288] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1755) [2025-01-08 00:52:10.237315] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:10.237341] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=138] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=849, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:10.237604] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BA8-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:10.237742] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BA8-0-0] [lt=135] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4941, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:10.237834] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:10.240143] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A7D-0-0] [lt=47] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:10.240394] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=159] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=4198, avg_group_log_size=349) [2025-01-08 00:52:10.241065] INFO [LIB] stat (utility.h:1210) [58690][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BA9-0-0] [lt=110] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:10.241097] INFO [LIB] stat (utility.h:1210) [58690][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BA9-0-0] [lt=31] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6064, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:10.241109] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58690][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BA9-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=104, avg_log_submit_to_flush_time=3563, avg_log_submit_to_slide_time=5960) [2025-01-08 00:52:10.241290] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1755, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:10.247061] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A7E-0-0] [lt=119] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:10.277854] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT WRITE LOG COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=1739, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:10.277915] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=58] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9932) [2025-01-08 00:52:10.277941] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=251, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:10.277974] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:10.279468] INFO [LIB] stat (utility.h:1210) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FFD-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:10.279495] INFO [LIB] stat (utility.h:1210) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FFD-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=3458, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:10.279507] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58666][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2FFD-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=15, avg_log_gen_to_submit_time=108, avg_log_submit_to_flush_time=2099, avg_log_submit_to_slide_time=3350) [2025-01-08 00:52:10.279653] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10521, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:10.284908] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=57] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:10.285770] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=24] ====== check clog disk timer task ====== [2025-01-08 00:52:10.285798] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:52:10.290103] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:52:10.290141] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.310254] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A7F-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:10.318659] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.318700] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.327307] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A80-0-0] [lt=51] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:10.327756] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C4D-0-0] [lt=34] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:10.327820] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C4D-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:52:10.327886] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C4D-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=14, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:10.327899] INFO [LIB] stat (utility.h:1210) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C4D-0-0] [lt=12] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=16952, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:10.327907] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C4D-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=2342, avg_log_submit_to_slide_time=15691) [2025-01-08 00:52:10.327978] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C4D-0-0] [lt=68][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18500, log_task={header:{begin_lsn:{lsn:2251715}, end_lsn:{lsn:2251837}, log_id:18500, min_scn:{val:1736326329856767003, v:0}, max_scn:{val:1736326329856767003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2251593}, prev_proposal_id:1, committed_end_lsn:{lsn:2251715}, data_checksum:940009244, accum_checksum:3782365038, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326330226328, freeze_ts:1736326330226328, submit_ts:1736326330226340, flushed_ts:1736326330231858, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:5518}, fs_cb_begin_ts=1736326330327871, log_life_time=101543) [2025-01-08 00:52:10.328116] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:10.328207] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18410, miss_cnt:0, cache_read_size:2251837, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:10.328735] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:52:10.329720] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2049, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:10.329761] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:52:10.329795] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=264, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:10.329958] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=79, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:10.336131] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:10.340917] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A81-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:10.343785] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=111] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:10.347198] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A82-0-0] [lt=40] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:52:10.349029] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=23] failed to get result(ret=-4008) [2025-01-08 00:52:10.349113] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=16] check new jobs(ret=0, tenant_id=1001, is_oracle_tenant=false, job_infos=[{tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326185605925, start_date_:1736325283345722, last_date_:1736326183353587, this_date_:0, next_date_:1736327083345722, end_date_:64060560000000000, total_:6596354, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736410783345628, last_date_:0, this_date_:0, next_date_:1736410783345628, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:52:10.349341] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=224] check all tenants(ret=0) [2025-01-08 00:52:10.351835] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] worker thread created(tenant_->id()=1002, group_id_=1, token=6) [2025-01-08 00:52:10.351837] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58702][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:10.362492] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=119] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326330362452], gts:1736326330361294000, latest_srr:[mts=1736326330362452]}) [2025-01-08 00:52:10.362564] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=53] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326330362547], gts:1736326330361294000, latest_srr:[mts=1736326330362547]}) [2025-01-08 00:52:10.362884] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326330362842], gts:1736326330361294000, latest_srr:[mts=1736326330362842]}) [2025-01-08 00:52:10.363324] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=18717696) [2025-01-08 00:52:10.379545] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1736326330379541, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007653, add_bytes: 0 [2025-01-08 00:52:10.386733] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:10.388427] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=38] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:10.390303] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.399029] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:10.413413] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:52:10.415434] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=87] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:10.417798] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=6] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:10.418601] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58693][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=57] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:52:10.434078] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1755, avg_group_log_size=159) [2025-01-08 00:52:10.434133] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1755) [2025-01-08 00:52:10.441821] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50560][T1001_TimerWK2][T1001][YB42C0A89F89-00062B2D1DE84E81-0-0] [lt=25] MediumCheckTask(cost_ts=9) [2025-01-08 00:52:10.442603] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4198) [2025-01-08 00:52:10.444866] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=138] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:10.445027] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=83] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, bw:49023, [delay/us]:prepare:6, schedule:339, submit:177, rt:1292, total:1816) [2025-01-08 00:52:10.445048] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:10.445113] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=49] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:49021, [delay/us]: prepare:6, schedule:339, submit:177, device:1292, total:1816) [2025-01-08 00:52:10.445127] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:10.445225] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4915, iops:15, bw:73528, [delay/us]:prepare:10, schedule:94, submit:86, rt:1198, total:1389) [2025-01-08 00:52:10.445238] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:10.445277] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4915, avg_iops:15, avg_bw:73529, [delay/us]: prepare:10, schedule:94, submit:86, device:1198, total:1389) [2025-01-08 00:52:10.445287] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:10.445359] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16340, [delay/us]:prepare:17, schedule:17, submit:0, rt:786, total:822) [2025-01-08 00:52:10.445389] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4193, iops:42, bw:175663, [delay/us]:prepare:9, schedule:113, submit:539, rt:1150, total:1812) [2025-01-08 00:52:10.445466] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=72] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:10.445503] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16341, [delay/us]: prepare:17, schedule:17, submit:0, device:786, total:822) [2025-01-08 00:52:10.445520] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4194, avg_iops:42, avg_bw:175669, [delay/us]: prepare:9, schedule:113, submit:539, device:1150, total:1812) [2025-01-08 00:52:10.445531] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:10.445551] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4085, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:298, total:306) [2025-01-08 00:52:10.445575] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:10.445602] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:298, total:306) [2025-01-08 00:52:10.445620] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:10.445635] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:10.445643] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:10.445650] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:10.445656] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=202kB/s, net_out=216kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:10.452554] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D1969110C-0-0] [lt=27] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.782375631332397461e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=891, Throttle Trigger(MB)=534, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.821863121442752842e+01) [2025-01-08 00:52:10.452850] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=68] get number(ret=0, service_type_=0, range=1, base_id=1736326330449697000, start_id=1736326330449697000, end_id=1736326330449697001) [2025-01-08 00:52:10.460984] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.461032] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.483767] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBC9-0-0] [lt=25][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193835397598, cache_obj->added_lc()=false, cache_obj->get_object_id()=13716, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:10.491114] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.513768] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43733714}], scn_array=[cnt:1, {val:1736326330498263000, v:0}], curr_size=589, accum_size=1767, time_cost=11069) [2025-01-08 00:52:10.519520] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=39] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.528953] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:52:10.529141] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58702][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C51-0-0] [lt=24] init point thread id with(&point=0x5572d50e7d00, idx_=1359, point=[thread id=58702, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58702) [2025-01-08 00:52:10.529836] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBCB-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193835443667, cache_obj->added_lc()=false, cache_obj->get_object_id()=13717, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:10.534852] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=36] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:312K/73:159795 disk:312K/73:159795 t1m2:52K/12:314354 t1m2g0:52K/12:314354 t1002m2:184K/45:133868 t1002m2g0:184K/45:133868 t1001m2:76K/16:116795 t1001m2g0:76K/16:116795 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:10.535133] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=128] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5342472704, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:52:10.537072] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=206] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:10.537769] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=27] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5342472704, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:52:10.541982] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=75] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.544091] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.545394] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBD0-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1736326330543482000, start_id=1736326330543482000, end_id=1736326330543482001) [2025-01-08 00:52:10.545549] INFO [LIB] stat (utility.h:1210) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBD0-0-0] [lt=39] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=102, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:10.545637] INFO [LIB] stat (utility.h:1210) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBD0-0-0] [lt=15] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:10.551804] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=2117, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:10.551876] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:10.551892] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=4312, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:10.551993] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:52:10.557312] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=28] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:52:10.589320] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8877, offset=2522779}, end_cursor:ObLogCursor{file_id=1, log_id=8878, offset=2523084}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894dac050, len:4453, log_data_len:305, seq:8877, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:10.591781] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326326578023, last_first_register_time_us=1736324454612308, register_time_us=1736326326578023, curr_time_us=1736326330591778) [2025-01-08 00:52:10.591860] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=47] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326330591829, last_submit_register_req_time_us=1736326330591829, register_parent_reason=1) [2025-01-08 00:52:10.591878] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:52:10.591931] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:52:10.591955] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.597168] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E82-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:10.597291] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E82-0-0] [lt=100] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326330361294001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326330361294001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:10.609844] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E82-0-0] [lt=83] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9933) [2025-01-08 00:52:10.619991] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2264742}], scn_array=[cnt:1, {val:1736326330385691002, v:0}], curr_size=122, accum_size=488, time_cost=23023) [2025-01-08 00:52:10.623871] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=57][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:43734303}], scn_array=[cnt:1, {val:1736326330609240000, v:0}], curr_size=1135, accum_size=2902, time_cost=14001) [2025-01-08 00:52:10.631469] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=42] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8878, offset=2523084}, end_cursor:ObLogCursor{file_id=1, log_id=8879, offset=2523381}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894db0050, len:4148, log_data_len:297, seq:8878, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:10.632435] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E82-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:10.632499] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E82-0-0] [lt=41] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326330361294001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326330361294001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:10.649439] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=61] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8879, offset=2523381}, end_cursor:ObLogCursor{file_id=1, log_id=8880, offset=2523678}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894db4050, len:3851, log_data_len:297, seq:8879, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:10.653637] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4518) [2025-01-08 00:52:10.654694] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=49] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8880, offset=2523678}, end_cursor:ObLogCursor{file_id=1, log_id=8881, offset=2523975}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894db8050, len:3554, log_data_len:297, seq:8880, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:10.655662] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=796, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:10.655700] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:10.666159] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=26] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:52:10.666306] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=130] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:52:10.666327] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=18] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=111) [2025-01-08 00:52:10.678004] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get number(ret=0, service_type_=0, range=1, base_id=1736326330674836000, start_id=1736326330674836000, end_id=1736326330674836001) [2025-01-08 00:52:10.688866] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D2248464F-0-0] [lt=190] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:52:10.692083] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.697403] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [46340][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=102] XA scheduler heartbeat task statistics(avg_time=0) [2025-01-08 00:52:10.705000] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2025-01-08 00:52:10.706528] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=101] PNIO [ratelimit] time: 1736326330706525, bytes: 888001951, bw: 0.246201 MB/s, add_ts: 1008369, add_bytes: 260321 [2025-01-08 00:52:10.710276] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=56] ====== tenant freeze timer task ====== [2025-01-08 00:52:10.726974] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C55-0-0] [lt=51][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18504, log_task={header:{begin_lsn:{lsn:2252203}, end_lsn:{lsn:2252325}, log_id:18504, min_scn:{val:1736326330355079002, v:0}, max_scn:{val:1736326330355079002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2252081}, prev_proposal_id:1, committed_end_lsn:{lsn:2252203}, data_checksum:940009244, accum_checksum:2186886464, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326330626317, freeze_ts:1736326330626317, submit_ts:1736326330626329, flushed_ts:1736326330631161, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:4832}, fs_cb_begin_ts=1736326330726958, log_life_time=100641) [2025-01-08 00:52:10.730082] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=28] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=31456, replace_num=15728) [2025-01-08 00:52:10.730159] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=75] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:52:10.737858] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:10.754249] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=42] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:52:10.763344] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:52:10.763414] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=54] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:52:10.763432] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:52:10.763440] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:52:10.763448] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:52:10.763456] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=99) [2025-01-08 00:52:10.771339] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=25] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:52:10.771456] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=111] start lib cache evict(tenant_id=1, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:52:10.771473] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=15] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=95862784, mem_limit=107374180, cache_obj_num=311, cache_node_num=311) [2025-01-08 00:52:10.771483] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:10.774170] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=10] Dumping All Cache Objs(alloc_obj_list.count()=311, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:13562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13566, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13572, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211185}, {obj_id:13582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225286}, {obj_id:13600, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180074}, {obj_id:13604, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206594}, {obj_id:13608, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185315}]) [2025-01-08 00:52:10.775031] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=857] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:10.775397] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=34] success to update min reserved snapshot(reserved_snapshot=1736324524430078000, duration=1800, snapshot_gc_ts=1736326324430078000) [2025-01-08 00:52:10.779483] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=37] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324524430078000) [2025-01-08 00:52:10.779511] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=25] update reserved snapshot finished(cost_ts=4121, reserved_snapshot=1736324524430078000) [2025-01-08 00:52:10.792256] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.796102] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB62B-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:10.796154] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB62B-0-0] [lt=32] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326330355079003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326330355079003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:10.813723] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=87] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:52:10.838948] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64] worker thread created(tenant_->id()=1002, group_id_=1, token=7) [2025-01-08 00:52:10.838941] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58704][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:10.839245] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BBB-0-0] [lt=86] init point thread id with(&point=0x5572d50e7d40, idx_=1360, point=[thread id=58704, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58704) [2025-01-08 00:52:10.840657] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=63] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-01-08 00:52:10.841060] INFO pn_ratelimit (group.c:653) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:52:10.843979] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=48] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:10.845577] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58701][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5DFC-0-0] [lt=26] init point thread id with(&point=0x5572d50e7d80, idx_=1361, point=[thread id=58701, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58701) [2025-01-08 00:52:10.856202] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=36] gc stale ls task succ [2025-01-08 00:52:10.856586] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=79] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=651907072, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=421834752, wash_ratio=6, tenant_id=1001) [2025-01-08 00:52:10.856669] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=46] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:52:10.856690] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=18] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2909, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:52:10.856802] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1114390528, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2106834944, wash_ratio=6, tenant_id=1002) [2025-01-08 00:52:10.856884] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=49] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:52:10.856899] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46100][TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:52:10.859098] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:10.859130] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-01-08 00:52:10.859141] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:10.859155] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:10.859169] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193834772776, flush_io_finished_ret_:0, flush_io_finished_round_:1993, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:10.863513] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=149] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=18717696) [2025-01-08 00:52:10.864727] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21] ts response task statistics(alloc_count=61294, free_count=30646) [2025-01-08 00:52:10.864771] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:10.867537] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=71] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:10.872619] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5343327847, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:10.872661] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:52:10.872674] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8bf5f02b0) [2025-01-08 00:52:10.872683] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5343327847, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:10.877542] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC455-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:10.887048] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=32] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:10.889934] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE8-0-0] [lt=15] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:10.890098] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE8-0-0] [lt=160] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:10.890417] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:10.891084] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=70] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2252625}, scn={val:1736326330355079005, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:52:10.891126] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:32, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326330890908, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2252681}, next_to_submit_scn_:{val:1736326330355079006, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2252569}, end_lsn:{lsn:2252681}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2252681}, readable_log_tail:{lsn:2252681}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64852087, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326330355079005, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326330355079005, v:0}, data_checksum:1159443327, flag:30429}}, init_mode_version:0, accumulate_checksum:2052192296, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326330355079005, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18417, miss_cnt:0, cache_read_size:2252681, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2252625}, scn_:{val:1736326330355079005, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326330916168, first_handle_ts_:1736326330891124, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad96b004150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:52:10.891546] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324525712447000) [2025-01-08 00:52:10.891574] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2252625}, scn_:{val:1736326330355079005, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326330916168, first_handle_ts_:1736326330891530, replay_cost_:42, retry_cost_:-1, read_log_buf_:0x2ad96b004160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326330890908, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2252681}, next_to_submit_scn_:{val:1736326330355079006, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2252569}, end_lsn:{lsn:2252681}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2252681}, readable_log_tail:{lsn:2252681}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64852087, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326330355079005, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326330355079005, v:0}, data_checksum:1159443327, flag:30429}}, init_mode_version:0, accumulate_checksum:2052192296, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326330355079005, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18417, miss_cnt:0, cache_read_size:2252681, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=8, ret=0) [2025-01-08 00:52:10.892379] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.897037] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.897075] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.897099] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.897499] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:10.899118] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D5-0-0] [lt=95] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:10.899236] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D5-0-0] [lt=48] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326330886616000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326330886616000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:10.916051] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:10.918612] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:10.920553] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=47] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18756663}, readable_log_tail:{lsn:18756663}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48348105, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:10.920655] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=101] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:52:10.920684] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2231331}, readable_log_tail:{lsn:2231331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64873437, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:10.920738] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=53] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:52:10.920750] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] get_replay_process success(id={id:1002}, submitted_log_size=2265230, unsubmitted_log_size=2265230, replayed_log_size=2265230, unreplayed_log_size=0) [2025-01-08 00:52:10.920760] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] get_replay_process success(id={id:1003}, submitted_log_size=2252681, unsubmitted_log_size=2252681, replayed_log_size=2252681, unreplayed_log_size=0) [2025-01-08 00:52:10.920787] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:52:10.930458] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=70] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:52:10.937768] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA2F-0-0] [lt=23][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8fc48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44909} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7530} write_epoch=1, prev=this=0x2ad8c1457368 trans_version={val:1736322641094761000, v:0} scn={val:1736322641094761000, v:0} tx_id={txid:40108} prev=(nil) next=(nil) modify_count=0 acc_checksum=1277335154 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:8294} write_epoch=1, snapshot_version={val:1736326330933278000, v:0}, *this={this=0x2ad8c14572d8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1457368 latest_compact_node=(nil) max_trans_version={val:1736326269979940000, v:0} max_trans_id=44825 max_elr_trans_version={val:1736326269979940000, v:0} max_elr_trans_id=44825 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641094761000, v:0} min_modify_scn={val:1736322641094761000, v:0}}) [2025-01-08 00:52:10.937834] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA2F-0-0] [lt=65][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8fc48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44909} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7530} write_epoch=1, ctx={this:0x2ad8b78d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326360832891, tablet_id:{id:309}, table_iter:0x2ad8b78d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326360832891, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326330933278000, v:0}, tx_id:{txid:44909}, scn:{branch:0, seq:3474}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44909}, tx_desc:{this:0x2ad8b7164ae0, tx_id:{txid:44909}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326330937284, commit_ts:-1, finish_ts:-1, timeout_us:29999974, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331028917, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0434d0, mem_ctx:0x2ad89c044a50, tx_scn:{branch:0, seq:7530}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:10.937930] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA2F-0-0] [lt=93][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44909}, data:0x2ad8b78d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:7530}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:10.937948] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA2F-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44909} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:10.937980] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA2F-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:10.937992] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA2F-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad92a20c6a0, local_datums:0x2ad92a20c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad92a20c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad92a20c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c373, hex: 41C22BFA2D2B0600, int: 1736326330958401}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c37b, hex: 41C22BFA2D2B0600, int: 1736326330958401}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c383, hex: 01BCF90000000000, int: 16366593, num_digit0: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c38b, hex: 00C0720100000000, int: 24297472}]}) [2025-01-08 00:52:10.938150] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA2F-0-0] [lt=105][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:10.938654] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:10.939017] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=149] [CACHE] tenant_id= 1 cache memory info: [CACHE] tenant_id= 1 | cache_name= schema_cache | cache_size= 54098944 | cache_store_size= 52019200 | cache_map_size= 2079744 | kv_cnt= 3298 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tablet_table_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 10 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= vtable_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 54 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1176 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_block_cache | cache_size= 20806656 | cache_store_size= 18726912 | cache_map_size= 2079744 | kv_cnt= 2397 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 22 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= bf_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 3 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= fuse_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 56 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= storage_meta_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 1126 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 8 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= log_kv_cache | cache_size= 89472000 | cache_store_size= 87392256 | cache_map_size= 2079744 | kv_cnt= 1289 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 191 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 275 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_ds_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 31 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-01-08 00:52:10.939151] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=38] [CACHE] tenant_id= 1001 cache memory info: [CACHE] tenant_id= 1001 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 989 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1514 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= storage_meta_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 992 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= log_kv_cache | cache_size= 47856640 | cache_store_size= 45776896 | cache_map_size= 2079744 | kv_cnt= 667 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 58 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 242 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-01-08 00:52:10.939271] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=17] [CACHE] tenant_id= 1002 cache memory info: [CACHE] tenant_id= 1002 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 710 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1066 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 712 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= log_kv_cache | cache_size= 24968192 | cache_store_size= 22888448 | cache_map_size= 2079744 | kv_cnt= 320 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 47 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 234 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-01-08 00:52:10.939294] INFO print_current_status (ob_kvcache_hazard_version.cpp:328) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=11] [KVCACHE-HAZARD] hazard version status info: current_version: 11275 | min_version= 11275 | total_nodes_count: 0 | +08:00READ-COMMITTEDOceanBase_CE 4.3.5.0 (r100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5) (Built Dec 31 2024 17:35:01)/ob/data-08:00AUTOmysql_native_passwordYESNOYESen_US [2025-01-08 00:52:10.939423] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46386][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA2F-0-0] [lt=23] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10904}, to_seq_no={branch:0, seq:3475}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44909} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:10.939505] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:152) [46386][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA2F-0-0] [lt=50] transaction rpc statistics(total_rt=2614, total_process=18, avg_rt=137) [2025-01-08 00:52:10.940451] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58705][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:10.940667] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=140] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:52:10.957055] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA31-0-0] [lt=27][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963664608 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44910} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:4937} write_epoch=1, prev=this=0x2ad8dae1b7e8 trans_version={val:1736322641115540001, v:0} scn={val:1736322641115540001, v:0} tx_id={txid:40109} prev=(nil) next=(nil) modify_count=0 acc_checksum=2685977296 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:10458} write_epoch=1, snapshot_version={val:1736326330954575000, v:0}, *this={this=0x2ad8dae1b758 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae1b7e8 latest_compact_node=(nil) max_trans_version={val:1736326270052653001, v:0} max_trans_id=44826 max_elr_trans_version={val:1736326270052653001, v:0} max_elr_trans_id=44826 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641115540001, v:0} min_modify_scn={val:1736322641115540001, v:0}}) [2025-01-08 00:52:10.957122] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA31-0-0] [lt=66][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963664608 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44910} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:4937} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326360854496, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326360854496, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326330954575000, v:0}, tx_id:{txid:44910}, scn:{branch:0, seq:4934}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44910}, tx_desc:{this:0x2ad8b0966b40, tx_id:{txid:44910}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326330955745, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331050614, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:4937}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:10.957291] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA31-0-0] [lt=166][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44910}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4937}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:10.957311] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA31-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44910} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:10.957343] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA31-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:10.957355] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA31-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c375, hex: 10152CFA2D2B0600, int: 1736326330979600}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37d, hex: 10152CFA2D2B0600, int: 1736326330979600}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:10.957462] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA31-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:10.958482] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA31-0-0] [lt=13] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:4940}, to_seq_no={branch:0, seq:4935}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44910} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:10.964748] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46503][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA32-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326330963789000, start_id=1736326330963789000, end_id=1736326330963789001) [2025-01-08 00:52:10.975376] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA33-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963664880 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44911} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4800} write_epoch=1, prev=this=0x2ad8c1563780 trans_version={val:1736324386058996000, v:0} scn={val:1736324386058996000, v:0} tx_id={txid:42211} prev=(nil) next=(nil) modify_count=0 acc_checksum=3597167689 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:12192} write_epoch=1, snapshot_version={val:1736326330973497000, v:0}, *this={this=0x2ad8c15636f0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1563780 latest_compact_node=(nil) max_trans_version={val:1736326270139259000, v:0} max_trans_id=44827 max_elr_trans_version={val:1736326270139259000, v:0} max_elr_trans_id=44827 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386058996000, v:0} min_modify_scn={val:1736324386058996000, v:0}}) [2025-01-08 00:52:10.975450] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA33-0-0] [lt=73][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963664880 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44911} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4800} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326360873415, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326360873415, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326330973497000, v:0}, tx_id:{txid:44911}, scn:{branch:0, seq:4797}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44911}, tx_desc:{this:0x2ad8b0970d20, tx_id:{txid:44911}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326330974658, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331068103, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:4800}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:10.975542] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA33-0-0] [lt=90][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44911}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4800}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:10.975559] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA33-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44911} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:10.975595] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA33-0-0] [lt=33][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:10.975608] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA33-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c372, hex: 6C672CFA2D2B0600, int: 1736326331000684}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37a, hex: 6C672CFA2D2B0600, int: 1736326331000684}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:10.975769] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA33-0-0] [lt=116][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:10.976832] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA33-0-0] [lt=37] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:4803}, to_seq_no={branch:0, seq:4798}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44911} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:10.991691] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA35-0-0] [lt=82][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8da4c3f10 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44912} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:2437} write_epoch=1, prev=this=0x2ad8c1563968 trans_version={val:1736324386116687000, v:0} scn={val:1736324386116687000, v:0} tx_id={txid:42212} prev=(nil) next=(nil) modify_count=0 acc_checksum=1789153443 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4159} write_epoch=1, snapshot_version={val:1736326330988963000, v:0}, *this={this=0x2ad8c15638d8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1563968 latest_compact_node=(nil) max_trans_version={val:1736326270185992000, v:0} max_trans_id=44828 max_elr_trans_version={val:1736326270185992000, v:0} max_elr_trans_id=44828 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386116687000, v:0} min_modify_scn={val:1736324386116687000, v:0}}) [2025-01-08 00:52:10.991826] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA35-0-0] [lt=133][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8da4c3f10 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44912} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:2437} write_epoch=1, ctx={this:0x2ad8b79cf678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326360889597, tablet_id:{id:309}, table_iter:0x2ad8b79cdb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326360889597, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326330988963000, v:0}, tx_id:{txid:44912}, scn:{branch:0, seq:2434}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44912}, tx_desc:{this:0x2ad8b5d4c660, tx_id:{txid:44912}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326330991142, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331086657, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:2437}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:10.991926] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA35-0-0] [lt=97][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44912}, data:0x2ad8b79cc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:2437}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:10.991944] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA35-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44912} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:10.991978] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA35-0-0] [lt=31][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:10.991992] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA35-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96860c6a0, local_datums:0x2ad96860c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96860c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96860c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96860c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96860c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96860c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96860c372, hex: 239E2CFA2D2B0600, int: 1736326331014691}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96860c37a, hex: 239E2CFA2D2B0600, int: 1736326331014691}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96860c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96860c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:10.992157] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46494][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA35-0-0] [lt=105][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:10.992463] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:10.994102] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46501][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA35-0-0] [lt=94] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:2440}, to_seq_no={branch:0, seq:2435}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44912} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:10.998928] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58705][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBDA-0-0] [lt=27] init point thread id with(&point=0x5572d50e7dc0, idx_=1362, point=[thread id=58705, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58705) [2025-01-08 00:52:11.001814] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684BCB-0-0] [lt=31] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:11.011496] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA37-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963664af8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44913} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=1, prev=this=0x2ad96365ef58 trans_version={val:1736326270208368000, v:0} scn={val:1736326270208368000, v:0} tx_id={txid:44829} prev=0x2ad8dae8a5e0 next=(nil) modify_count=120 acc_checksum=827550617 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:20342} write_epoch=3, snapshot_version={val:1736326331009341000, v:0}, *this={this=0x2ad8c14578a8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad96365ef58 latest_compact_node=0x2ad8dae8a5e0 max_trans_version={val:1736326270208368000, v:0} max_trans_id=44829 max_elr_trans_version={val:1736326270208368000, v:0} max_elr_trans_id=44829 latest_compact_ts=1736326270222032 last_compact_cnt=119 total_trans_node_cnt=121 max_modify_scn={val:1736326270208368000, v:0} min_modify_scn={val:1736322641129989000, v:0}}) [2025-01-08 00:52:11.011624] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA37-0-0] [lt=127][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963664af8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44913} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326360909794, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326360909794, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331009341000, v:0}, tx_id:{txid:44913}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44913}, tx_desc:{this:0x2ad8b097cf60, tx_id:{txid:44913}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331010469, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331106929, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:7}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.011720] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA37-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44913}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:7}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.011736] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA37-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44913} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.011768] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA37-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.011779] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA37-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: CCF02CFA2D2B0600, int: 1736326331035852}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: CCF02CFA2D2B0600, int: 1736326331035852}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 64EB090900000000, int: 151645028}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 64EB090900000000, int: 151645028}]}) [2025-01-08 00:52:11.011910] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA37-0-0] [lt=85][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.013088] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA37-0-0] [lt=42] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44913} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.013641] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF79-0-0] [lt=33] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326331115525, keepalive_ts:-1}) [2025-01-08 00:52:11.013671] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF79-0-0] [lt=29] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326331115525, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:11.033616] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA39-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963664f50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44914} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:2306} write_epoch=1, prev=this=0x2ad8c1564190 trans_version={val:1736324386214262001, v:0} scn={val:1736324386214262001, v:0} tx_id={txid:42214} prev=(nil) next=(nil) modify_count=0 acc_checksum=3172668830 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3645} write_epoch=1, snapshot_version={val:1736326331029999000, v:0}, *this={this=0x2ad8c1564100 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1564190 latest_compact_node=(nil) max_trans_version={val:1736326270251301000, v:0} max_trans_id=44830 max_elr_trans_version={val:1736326270251301000, v:0} max_elr_trans_id=44830 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386214262001, v:0} min_modify_scn={val:1736324386214262001, v:0}}) [2025-01-08 00:52:11.033702] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA39-0-0] [lt=85][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963664f50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44914} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:2306} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326360930503, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326360930503, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331029999000, v:0}, tx_id:{txid:44914}, scn:{branch:0, seq:2303}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44914}, tx_desc:{this:0x2ad8b0981020, tx_id:{txid:44914}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331032985, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331130307, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:2306}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.033877] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA39-0-0] [lt=171][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44914}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:2306}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.033900] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA39-0-0] [lt=23][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44914} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.033933] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA39-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.033944] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA39-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 03452DFA2D2B0600, int: 1736326331057411}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 03452DFA2D2B0600, int: 1736326331057411}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 710B080000000000, int: 527217}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00D03D0000000000, int: 4050944}]}) [2025-01-08 00:52:11.034081] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA39-0-0] [lt=83][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.035540] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=64, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:11.035585] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:52:11.035604] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1332) [2025-01-08 00:52:11.035616] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:11.035561] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46507][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA39-0-0] [lt=6] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:2309}, to_seq_no={branch:0, seq:2304}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44914} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.038538] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1210) [2025-01-08 00:52:11.047539] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9279) [2025-01-08 00:52:11.049304] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58681][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6A9C-0-0] [lt=22] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=17, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=103, avg_log_submit_to_flush_time=2276, avg_log_submit_to_slide_time=3839) [2025-01-08 00:52:11.049661] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=93] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9279, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:11.049740] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58666][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBD0-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326331151358, keepalive_ts:-1}) [2025-01-08 00:52:11.049811] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58666][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBD0-0-0] [lt=69] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326331151358, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:52:11.053088] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:52:11.054714] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58681][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=144] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:52:11.055643] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46387][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3B-0-0] [lt=41][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8da4c4188 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44915} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4788} write_epoch=1, prev=this=0x2ad8c1573950 trans_version={val:1736324508824146000, v:0} scn={val:1736324508824146000, v:0} tx_id={txid:42418} prev=0x2ad8c1573798 next=(nil) modify_count=4 acc_checksum=3409534941 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:18565} write_epoch=3, snapshot_version={val:1736326331052176000, v:0}, *this={this=0x2ad8c15646a8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8c1573950 latest_compact_node=(nil) max_trans_version={val:1736326270292568000, v:0} max_trans_id=44831 max_elr_trans_version={val:1736326270292568000, v:0} max_elr_trans_id=44831 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324508824146000, v:0} min_modify_scn={val:1736324386269257000, v:0}}) [2025-01-08 00:52:11.055722] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46387][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3B-0-0] [lt=78][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8da4c4188 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44915} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4788} write_epoch=1, ctx={this:0x2ad8ad751678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326360953266, tablet_id:{id:309}, table_iter:0x2ad8ad74fb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326360953266, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331052176000, v:0}, tx_id:{txid:44915}, scn:{branch:0, seq:3573}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44915}, tx_desc:{this:0x2ad8b5e2d120, tx_id:{txid:44915}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331054485, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331148040, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4788}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.055841] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46387][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3B-0-0] [lt=118][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44915}, data:0x2ad8ad74e508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4788}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.055862] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46387][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3B-0-0] [lt=21][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44915} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.055893] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46387][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3B-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.055904] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46387][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3B-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8d3c0c6a0, local_datums:0x2ad8d3c0c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8d3c0c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8d3c0c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c373, hex: A27B2DFA2D2B0600, int: 1736326331071394}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c37b, hex: A27B2DFA2D2B0600, int: 1736326331071394}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c383, hex: 1ED9080000000000, int: 579870}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c38b, hex: 00802F0000000000, int: 3112960}]}) [2025-01-08 00:52:11.056019] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46387][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3B-0-0] [lt=83][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.059437] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3B-0-0] [lt=41] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8944}, to_seq_no={branch:0, seq:3574}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44915} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.073560] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3D-0-0] [lt=81][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad9636651d8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44916} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:2261} write_epoch=1, prev=this=0x2ad8c15649e0 trans_version={val:1736324386317997000, v:0} scn={val:1736324386317997000, v:0} tx_id={txid:42216} prev=(nil) next=(nil) modify_count=0 acc_checksum=866976763 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:8267} write_epoch=1, snapshot_version={val:1736326331067727000, v:0}, *this={this=0x2ad8c1564950 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c15649e0 latest_compact_node=(nil) max_trans_version={val:1736326270320752000, v:0} max_trans_id=44832 max_elr_trans_version={val:1736326270320752000, v:0} max_elr_trans_id=44832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386317997000, v:0} min_modify_scn={val:1736324386317997000, v:0}}) [2025-01-08 00:52:11.073671] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3D-0-0] [lt=75][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad9636651d8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44916} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:2261} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326360971266, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326360971266, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331067727000, v:0}, tx_id:{txid:44916}, scn:{branch:0, seq:2258}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44916}, tx_desc:{this:0x2ad8b0983080, tx_id:{txid:44916}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331067727, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331168382, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:2261}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.073766] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3D-0-0] [lt=128][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44916}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:2261}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.073783] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3D-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44916} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.073814] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3D-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.073825] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3D-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c375, hex: 80E02DFA2D2B0600, int: 1736326331097216}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37d, hex: 80E02DFA2D2B0600, int: 1736326331097216}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:11.074046] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3D-0-0] [lt=173][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.078200] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=105] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=70, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:11.078294] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=77] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:11.078631] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CD0-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:11.080245] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3D-0-0] [lt=41] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:2264}, to_seq_no={branch:0, seq:2259}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44916} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.080627] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9384) [2025-01-08 00:52:11.082434] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CD0-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:11.092578] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.094364] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49BD-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:11.094523] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49BD-0-0] [lt=151] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:11.101305] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3F-0-0] [lt=15][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963665468 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44917} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:5548} write_epoch=1, prev=this=0x2ad8c1564bd8 trans_version={val:1736324386348542000, v:0} scn={val:1736324386348542000, v:0} tx_id={txid:42217} prev=(nil) next=(nil) modify_count=0 acc_checksum=836285912 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:3791} write_epoch=1, snapshot_version={val:1736326331096579000, v:0}, *this={this=0x2ad8c1564b48 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1564bd8 latest_compact_node=(nil) max_trans_version={val:1736326270384196000, v:0} max_trans_id=44833 max_elr_trans_version={val:1736326270384196000, v:0} max_elr_trans_id=44833 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386348542000, v:0} min_modify_scn={val:1736324386348542000, v:0}}) [2025-01-08 00:52:11.101388] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3F-0-0] [lt=82][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963665468 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44917} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no={branch:0, seq:5548} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326360995904, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326360995904, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331096579000, v:0}, tx_id:{txid:44917}, scn:{branch:0, seq:5545}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44917}, tx_desc:{this:0x2ad8b0993380, tx_id:{txid:44917}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331100830, commit_ts:-1, finish_ts:-1, timeout_us:29999979, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331193615, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:5548}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.101487] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3F-0-0] [lt=97][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44917}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:5548}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.101504] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3F-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44917} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.101560] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3F-0-0] [lt=52][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.101577] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3F-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c375, hex: 3A3C2EFA2D2B0600, int: 1736326331120698}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37d, hex: 3A3C2EFA2D2B0600, int: 1736326331120698}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:11.101764] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA3F-0-0] [lt=129][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.103044] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA3F-0-0] [lt=54] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7852}, to_seq_no={branch:0, seq:5546}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44917} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.107021] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=59] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9973, avg_group_log_size=623) [2025-01-08 00:52:11.110668] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326330874888000, v:0}, new_max_version={val:1736326330874888000, v:0}, last_min_version={val:1736326330702577999, v:0}, last_max_version={val:1736326330702577999, v:0}, total_time_us=5793, delta_us=235778) [2025-01-08 00:52:11.110718] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326330874888000, v:0}, min_version={val:1736326330874888000, v:0}, max_version={val:1736326330874888000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:11.112981] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46383][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA40-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326331112260000, start_id=1736326331112260000, end_id=1736326331112260001) [2025-01-08 00:52:11.127798] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58705][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C5D-0-0] [lt=54] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:52:11.130749] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:52:11.133620] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA43-0-0] [lt=18][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad9636658c0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44919} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:6844} write_epoch=1, prev=this=0x2ad96365fff8 trans_version={val:1736326270421100000, v:0} scn={val:1736326270421100000, v:0} tx_id={txid:44835} prev=0x2ad96365fe40 next=(nil) modify_count=62 acc_checksum=176231898 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:8641} write_epoch=3, snapshot_version={val:1736326331131700000, v:0}, *this={this=0x2ad8c15652e8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad96365fff8 latest_compact_node=0x2ad8dae89a00 max_trans_version={val:1736326270421100000, v:0} max_trans_id=44835 max_elr_trans_version={val:1736326270421100000, v:0} max_elr_trans_id=44835 latest_compact_ts=1736326209367843 last_compact_cnt=59 total_trans_node_cnt=63 max_modify_scn={val:1736326270421100000, v:0} min_modify_scn={val:1736324386409746000, v:0}}) [2025-01-08 00:52:11.133701] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA43-0-0] [lt=79][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad9636658c0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44919} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:6844} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361031400, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361031400, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331131700000, v:0}, tx_id:{txid:44919}, scn:{branch:0, seq:5710}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44919}, tx_desc:{this:0x2ad8b0997440, tx_id:{txid:44919}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331132839, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331227244, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:6844}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.133827] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA43-0-0] [lt=123][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44919}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:6844}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.133849] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA43-0-0] [lt=21][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44919} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.133882] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA43-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.133892] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA43-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 23C82EFA2D2B0600, int: 1736326331156515}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 23C82EFA2D2B0600, int: 1736326331156515}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 0C799B0600000000, int: 110852364}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 0C799B0600000000, int: 110852364}]}) [2025-01-08 00:52:11.134124] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA43-0-0] [lt=124][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.135863] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA43-0-0] [lt=72] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6847}, to_seq_no={branch:0, seq:5711}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44919} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.140147] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=42] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:11.140858] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6731165696, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:11.141953] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF4-0-0] [lt=25] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:52:11.141985] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CF4-0-0] [lt=25] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=34) [2025-01-08 00:52:11.157243] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA45-0-0] [lt=37][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963665d18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44920} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4608} write_epoch=1, prev=this=0x2ad963660280 trans_version={val:1736326270473273000, v:0} scn={val:1736326270473273000, v:0} tx_id={txid:44836} prev=0x2ad9636600c8 next=(nil) modify_count=62 acc_checksum=1693648954 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:22511} write_epoch=3, snapshot_version={val:1736326331155430000, v:0}, *this={this=0x2ad8c15656d0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad963660280 latest_compact_node=0x2ad8dae89c88 max_trans_version={val:1736326270473273000, v:0} max_trans_id=44836 max_elr_trans_version={val:1736326270473273000, v:0} max_elr_trans_id=44836 latest_compact_ts=1736326209421423 last_compact_cnt=59 total_trans_node_cnt=63 max_modify_scn={val:1736326270473273000, v:0} min_modify_scn={val:1736324386439933000, v:0}}) [2025-01-08 00:52:11.157308] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA45-0-0] [lt=64][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963665d18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44920} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:4608} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361055182, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361055182, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331155430000, v:0}, tx_id:{txid:44920}, scn:{branch:0, seq:4605}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44920}, tx_desc:{this:0x2ad8b099f5c0, tx_id:{txid:44920}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331156816, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331250313, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:4608}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.157445] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA45-0-0] [lt=134][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44920}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4608}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.157464] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA45-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44920} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.157496] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA45-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.157508] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA45-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 542A2FFA2D2B0600, int: 1736326331181652}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 542A2FFA2D2B0600, int: 1736326331181652}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 7170851100000000, int: 293957745}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 7170851100000000, int: 293957745}]}) [2025-01-08 00:52:11.157602] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA45-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.158628] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=74] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:52:11.158984] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:52:11.159467] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA45-0-0] [lt=56] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6886}, to_seq_no={branch:0, seq:4606}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44920} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.163258] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1003}, {id:1}]) [2025-01-08 00:52:11.163578] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D0EF-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:11.169953] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:11.170004] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:11.173324] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:11.173448] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=10] gc stale ls task succ [2025-01-08 00:52:11.176031] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB871D2-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:11.176147] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=88][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:11.176161] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:11.176623] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get_max_applied_scn(scn={val:1736326330863916001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2231697}, palf_committed_end_scn_:{val:1736326330863916002, v:0}, last_check_scn_:{val:1736326330863916002, v:0}, max_applied_cb_scn_:{val:1736326330863916001, v:0}}) [2025-01-08 00:52:11.176655] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2231697}, readable_log_tail:{lsn:2231697}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64873071, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:11.187488] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA47-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963665ee8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44921} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3557} write_epoch=1, prev=this=0x2ad8c1457f08 trans_version={val:1736322641150641000, v:0} scn={val:1736322641150641000, v:0} tx_id={txid:40112} prev=(nil) next=(nil) modify_count=0 acc_checksum=221902601 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3015} write_epoch=1, snapshot_version={val:1736326331185508000, v:0}, *this={this=0x2ad8c1457e78 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1457f08 latest_compact_node=(nil) max_trans_version={val:1736326270491010000, v:0} max_trans_id=44837 max_elr_trans_version={val:1736326270491010000, v:0} max_elr_trans_id=44837 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641150641000, v:0} min_modify_scn={val:1736322641150641000, v:0}}) [2025-01-08 00:52:11.187560] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA47-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963665ee8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44921} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3557} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361083442, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361083442, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331185508000, v:0}, tx_id:{txid:44921}, scn:{branch:0, seq:3554}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44921}, tx_desc:{this:0x2ad8b09a3680, tx_id:{txid:44921}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331186726, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331278597, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:3557}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.187659] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA47-0-0] [lt=97][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44921}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:3557}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.187677] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA47-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44921} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.187710] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA47-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.187723] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA47-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c374, hex: E6912FFA2D2B0600, int: 1736326331208166}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37c, hex: E6912FFA2D2B0600, int: 1736326331208166}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c384, hex: 0080420200000000, int: 37912576}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38c, hex: 0080420200000000, int: 37912576}]}) [2025-01-08 00:52:11.187876] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA47-0-0] [lt=58][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.189048] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA47-0-0] [lt=39] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:3560}, to_seq_no={branch:0, seq:3555}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44921} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.192834] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.193911] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:11.196602] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49BF-0-0] [lt=105] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:11.198778] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=107] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3511, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:11.198814] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:11.198859] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=170, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:11.199023] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=69, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:11.200725] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBDE-0-0] [lt=46] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:11.200751] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBDE-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13216, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:11.200811] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58704][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBDE-0-0] [lt=57] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=18, avg_log_submit_to_flush_time=3761, avg_log_submit_to_slide_time=13198) [2025-01-08 00:52:11.201019] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:11.201059] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18520, miss_cnt:0, cache_read_size:2265596, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:11.201439] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=20] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=236, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:11.201460] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=7, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:11.206161] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918545D-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:11.209737] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA49-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963666168 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44922} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:2612} write_epoch=1, prev=this=0x2ad8c14582e8 trans_version={val:1736322641160272000, v:0} scn={val:1736322641160272000, v:0} tx_id={txid:40113} prev=(nil) next=(nil) modify_count=0 acc_checksum=4054987080 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:1932} write_epoch=1, snapshot_version={val:1736326331207677000, v:0}, *this={this=0x2ad8c1458258 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c14582e8 latest_compact_node=(nil) max_trans_version={val:1736326270548894000, v:0} max_trans_id=44838 max_elr_trans_version={val:1736326270548894000, v:0} max_elr_trans_id=44838 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641160272000, v:0} min_modify_scn={val:1736322641160272000, v:0}}) [2025-01-08 00:52:11.209810] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA49-0-0] [lt=71][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963666168 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44922} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:2612} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361106542, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361106542, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331207677000, v:0}, tx_id:{txid:44922}, scn:{branch:0, seq:2609}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44922}, tx_desc:{this:0x2ad8b09af8c0, tx_id:{txid:44922}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331208816, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331301263, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:2612}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.209963] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA49-0-0] [lt=150][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44922}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:2612}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.209983] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA49-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44922} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.210057] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA49-0-0] [lt=71][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.210070] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA49-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 06ED2FFA2D2B0600, int: 1736326331231494}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 06ED2FFA2D2B0600, int: 1736326331231494}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 0050020000000000, int: 151552}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 0050020000000000, int: 151552}]}) [2025-01-08 00:52:11.210165] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA49-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.211324] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA49-0-0] [lt=109] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:2615}, to_seq_no={branch:0, seq:2610}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44922} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.224755] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:52:11.224895] WDIAG read_multi_row (ob_table_access_helper.h:238) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:52:11.226377] INFO delete_row (ob_table_access_helper.h:339) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326331224913) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326331224913) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:52:11.226471] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=40] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:52:11.226548] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=31] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:52:11.226629] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=33] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:52:11.235769] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=82, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:11.235811] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:11.238237] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2331, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:11.238307] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=68] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:11.238330] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=185, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:11.238362] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=206] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=11, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:11.238598] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=81] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2568, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:11.238623] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:11.238654] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=171, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:11.238725] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:11.238776] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:11.239279] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A97FD-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=135, avg_log_submit_to_flush_time=2841, avg_log_submit_to_slide_time=4754) [2025-01-08 00:52:11.240328] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BC6-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:11.240362] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BC6-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4553, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:11.240420] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:11.241388] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4B-0-0] [lt=77][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad9636663e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44923} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:8309} write_epoch=1, prev=this=0x2ad8c1566038 trans_version={val:1736324387086570000, v:0} scn={val:1736324387086570000, v:0} tx_id={txid:42223} prev=(nil) next=(nil) modify_count=0 acc_checksum=3412027634 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:3556} write_epoch=1, snapshot_version={val:1736326331238072000, v:0}, *this={this=0x2ad8c1565fa8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1566038 latest_compact_node=(nil) max_trans_version={val:1736326270610802000, v:0} max_trans_id=44839 max_elr_trans_version={val:1736326270610802000, v:0} max_elr_trans_id=44839 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324387086570000, v:0} min_modify_scn={val:1736324387086570000, v:0}}) [2025-01-08 00:52:11.241467] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4B-0-0] [lt=78][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad9636663e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44923} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:8309} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361136768, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361136768, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331238072000, v:0}, tx_id:{txid:44923}, scn:{branch:0, seq:4470}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44923}, tx_desc:{this:0x2ad8b09b1920, tx_id:{txid:44923}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331240393, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331333813, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:8309}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.241569] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4B-0-0] [lt=100][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44923}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:8309}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.241586] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4B-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44923} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.241619] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4B-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.241631] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4B-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c374, hex: FC6630FA2D2B0600, int: 1736326331262716}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37c, hex: FC6630FA2D2B0600, int: 1736326331262716}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c384, hex: 00D0F70000000000, int: 16240640}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38c, hex: 00D0F70000000000, int: 16240640}]}) [2025-01-08 00:52:11.241737] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4B-0-0] [lt=54][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.243261] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA4B-0-0] [lt=92] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8313}, to_seq_no={branch:0, seq:4471}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44923} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.250511] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA4C-0-0] [lt=104] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=24, avg_log_batch_cnt=1, total_group_log_size=14320, avg_group_log_size=596) [2025-01-08 00:52:11.254296] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=47] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:52:11.259504] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4D-0-0] [lt=31][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963666668 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44924} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:3570} write_epoch=1, prev=this=0x2ad8dae42eb0 trans_version={val:1736324509014921000, v:0} scn={val:1736324509014921000, v:0} tx_id={txid:42427} prev=0x2ad8c1575460 next=(nil) modify_count=4 acc_checksum=3656646905 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326330863916000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:1468} write_epoch=3, snapshot_version={val:1736326331256450000, v:0}, *this={this=0x2ad8c1566418 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8dae42eb0 latest_compact_node=(nil) max_trans_version={val:1736326270641542001, v:0} max_trans_id=44840 max_elr_trans_version={val:1736326270641542001, v:0} max_elr_trans_id=44840 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324509014921000, v:0} min_modify_scn={val:1736324387129570000, v:0}}) [2025-01-08 00:52:11.259561] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4D-0-0] [lt=56][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963666668 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44924} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:94} seq_no={branch:0, seq:3570} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361157246, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361157246, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331256450000, v:0}, tx_id:{txid:44924}, scn:{branch:0, seq:1150}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44924}, tx_desc:{this:0x2ad8b09a56e0, tx_id:{txid:44924}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331258532, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331356243, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:3570}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.259651] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4D-0-0] [lt=88][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44924}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:3570}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.259730] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4D-0-0] [lt=76][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44924} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.259762] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4D-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.259816] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4D-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c374, hex: 23B730FA2D2B0600, int: 1736326331283235}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37c, hex: 23B730FA2D2B0600, int: 1736326331283235}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c384, hex: 00E0B60000000000, int: 11984896}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38c, hex: 00E0B60000000000, int: 11984896}]}) [2025-01-08 00:52:11.259963] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA4D-0-0] [lt=112][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.261450] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA4D-0-0] [lt=79] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:3573}, to_seq_no={branch:0, seq:1151}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44924} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.268149] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:52:11.268355] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=196] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:52:11.268374] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=14] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:52:11.268397] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=22] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:11.271772] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=36] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:52:11.272638] INFO [STORAGE.TRANS] rollback_to_savepoint (ob_trans_part_ctx.cpp:8738) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA4D-0-0] [lt=42] rollback succeed trivially(trans_id={txid:44924}, ls_id={id:1}, op_sn=3, to_scn={branch:0, seq:1151}, last_scn={branch:0, seq:1151}) [2025-01-08 00:52:11.274473] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=51] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:52:11.276527] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=40] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:52:11.276563] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=35] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:52:11.278120] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084884-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:52:11.278240] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084884-0-0] [lt=19] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:52:11.278259] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084884-0-0] [lt=17] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:52:11.281887] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=3374, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:11.281922] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=33] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9384) [2025-01-08 00:52:11.281946] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=440, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:11.281988] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=38] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=20, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:11.283500] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=12] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:52:11.285242] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=25] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:11.285928] INFO [LIB] stat (utility.h:1210) [58666][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3CA4-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=12, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:11.285953] INFO [LIB] stat (utility.h:1210) [58666][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3CA4-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=5453, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:11.285964] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58666][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3CA4-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=98, avg_log_submit_to_flush_time=3879, avg_log_submit_to_slide_time=5354) [2025-01-08 00:52:11.286105] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=100] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9384, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:11.286286] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=72] ====== check clog disk timer task ====== [2025-01-08 00:52:11.286306] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=14] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:52:11.288608] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=59] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:52:11.289593] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=117] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1001, is_allowed_broadcast=true) [2025-01-08 00:52:11.291297] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=54] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:52:11.291339] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:52:11.292946] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.293112] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084884-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:52:11.293258] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084884-0-0] [lt=38] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:52:11.293276] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084884-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:52:11.294886] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D1-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1736326331293101000, start_id=1736326331293101000, end_id=1736326331293101001) [2025-01-08 00:52:11.296483] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=33] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:52:11.297936] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46499][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA4E-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326331297193000, start_id=1736326331297193000, end_id=1736326331297193001) [2025-01-08 00:52:11.303525] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=157] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:52:11.304578] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=47] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:52:11.305367] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=45] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:52:11.305406] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=37] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:52:11.306895] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084884-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:52:11.307056] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084884-0-0] [lt=68] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:52:11.307074] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084884-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:52:11.307097] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=13] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:52:11.307739] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=638] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:11.307791] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=44] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:52:11.307800] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=8] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:11.307820] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084884-0-0] [lt=16] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:52:11.307838] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084884-0-0] [lt=18] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:52:11.307847] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=6] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:52:11.307875] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=27] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:11.307940] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084884-0-0] [lt=61] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:52:11.307955] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084884-0-0] [lt=14] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:52:11.307966] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=5] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:52:11.307971] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=5] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:52:11.308012] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084884-0-0] [lt=38] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:52:11.308030] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084884-0-0] [lt=17] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:52:11.309932] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=7] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:52:11.309974] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=39] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:52:11.309986] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=11] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:52:11.310018] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:52:11.310031] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:52:11.310039] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:52:11.310051] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:52:11.310066] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084884-0-0] [lt=14] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:52:11.311876] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D17DBBA4F-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1736326331311336000, start_id=1736326331311336000, end_id=1736326331311336001) [2025-01-08 00:52:11.316451] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=29] ====== check clog disk timer task ====== [2025-01-08 00:52:11.316488] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=28] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:52:11.319859] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.320113] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.320851] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA50-0-0] [lt=96] get number(ret=0, service_type_=0, range=1, base_id=1736326331320232000, start_id=1736326331320232000, end_id=1736326331320232001) [2025-01-08 00:52:11.325925] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D2-0-0] [lt=26][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963666ce0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55793} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:5741} write_epoch=1, prev=this=0x2ad8c146c068 trans_version={val:1736322701346044000, v:0} scn={val:1736322701346044000, v:0} tx_id={txid:50196} prev=0x2ad8c146beb0 next=(nil) modify_count=2 acc_checksum=1214755906 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:18} write_epoch=3, snapshot_version={val:1736326331322917000, v:0}, *this={this=0x2ad8c14574c8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad8c146c068 latest_compact_node=(nil) max_trans_version={val:1736326270052653000, v:0} max_trans_id=55694 max_elr_trans_version={val:1736326270052653000, v:0} max_elr_trans_id=55694 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=3 max_modify_scn={val:1736322701346044000, v:0} min_modify_scn={val:1736322641103233000, v:0}}) [2025-01-08 00:52:11.326006] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D2-0-0] [lt=80][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963666ce0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55793} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:5741} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361220869, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361220869, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331322917000, v:0}, tx_id:{txid:55793}, scn:{branch:0, seq:5738}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55793}, tx_desc:{this:0x2ad8b09b7a40, tx_id:{txid:55793}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331325148, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331420572, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:5741}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.326097] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D2-0-0] [lt=90][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55793}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:5741}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.326145] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D2-0-0] [lt=46][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55793} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.326240] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D2-0-0] [lt=92][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.326252] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D2-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 2AD532FA2D2B0600, int: 1736326331421994}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 2AD532FA2D2B0600, int: 1736326331421994}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: A16FD80100000000, int: 30961569}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 0000730200000000, int: 41091072}]}) [2025-01-08 00:52:11.326418] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D2-0-0] [lt=118][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.328172] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D2-0-0] [lt=31] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8219}, to_seq_no={branch:0, seq:5739}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55793} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.328670] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C61-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:11.328765] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C61-0-0] [lt=82] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:52:11.329880] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1993, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:11.329939] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=457, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:11.330039] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=67, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:11.331183] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:52:11.335261] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA51-0-0] [lt=128][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963666eb0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44926} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:1113} write_epoch=1, prev=this=0x2ad8c14586c8 trans_version={val:1736322641178398000, v:0} scn={val:1736322641178398000, v:0} tx_id={txid:40115} prev=(nil) next=(nil) modify_count=0 acc_checksum=1913904902 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:2356} write_epoch=1, snapshot_version={val:1736326331331612000, v:0}, *this={this=0x2ad8c1458638 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c14586c8 latest_compact_node=(nil) max_trans_version={val:1736326270693129000, v:0} max_trans_id=44842 max_elr_trans_version={val:1736326270693129000, v:0} max_elr_trans_id=44842 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641178398000, v:0} min_modify_scn={val:1736322641178398000, v:0}}) [2025-01-08 00:52:11.335319] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA51-0-0] [lt=57][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963666eb0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44926} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:1113} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361231284, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361231284, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331331612000, v:0}, tx_id:{txid:44926}, scn:{branch:0, seq:1110}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44926}, tx_desc:{this:0x2ad8b09bdb60, tx_id:{txid:44926}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331334599, commit_ts:-1, finish_ts:-1, timeout_us:29999973, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331429971, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:1113}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.335412] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA51-0-0] [lt=91][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44926}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:1113}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.335430] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA51-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44926} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.335490] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA51-0-0] [lt=56][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.335503] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA51-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 70D231FA2D2B0600, int: 1736326331355760}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 70D231FA2D2B0600, int: 1736326331355760}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 00C0CF0000000000, int: 13615104}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00C0CF0000000000, int: 13615104}]}) [2025-01-08 00:52:11.335599] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA51-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.337615] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA51-0-0] [lt=110] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7816}, to_seq_no={branch:0, seq:1111}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44926} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.340580] INFO [LIB] stat (utility.h:1210) [58705][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BC9-0-0] [lt=49] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:11.340640] INFO [LIB] stat (utility.h:1210) [58705][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BC9-0-0] [lt=59] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4859, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:11.340885] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=99] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:11.341791] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6733262848, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:11.344182] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:11.351913] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46508][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D4-0-0] [lt=34][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963667088 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55794} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:4312} write_epoch=1, prev=this=0x2ad8c1457750 trans_version={val:1736322641116642000, v:0} scn={val:1736322641116642000, v:0} tx_id={txid:50112} prev=(nil) next=(nil) modify_count=0 acc_checksum=1602798291 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:11449} write_epoch=1, snapshot_version={val:1736326331348947000, v:0}, *this={this=0x2ad8c14576c0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1457750 latest_compact_node=(nil) max_trans_version={val:1736326270137671000, v:0} max_trans_id=55695 max_elr_trans_version={val:1736326270137671000, v:0} max_elr_trans_id=55695 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641116642000, v:0} min_modify_scn={val:1736322641116642000, v:0}}) [2025-01-08 00:52:11.351996] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46508][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D4-0-0] [lt=81][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963667088 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55794} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:88} seq_no={branch:0, seq:4312} write_epoch=1, ctx={this:0x2ad8b80d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361248797, tablet_id:{id:309}, table_iter:0x2ad8b80d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361248797, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331348947000, v:0}, tx_id:{txid:55794}, scn:{branch:0, seq:4309}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55794}, tx_desc:{this:0x2ad8b7183080, tx_id:{txid:55794}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331351481, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331447464, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:4312}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.352227] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46508][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D4-0-0] [lt=227][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55794}, data:0x2ad8b80d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4312}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.352272] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46508][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D4-0-0] [lt=43][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55794} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.352306] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46508][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D4-0-0] [lt=32][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.352317] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46508][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D4-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad93220c6a0, local_datums:0x2ad93220c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad93220c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad93220c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad93220c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad93220c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad93220c363, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad93220c375, hex: A93833FA2D2B0600, int: 1736326331447465}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad93220c37d, hex: A93833FA2D2B0600, int: 1736326331447465}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad93220c385, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad93220c38d, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:11.352547] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46508][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D4-0-0] [lt=124][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.353866] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46500][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D4-0-0] [lt=23] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7955}, to_seq_no={branch:0, seq:4310}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55794} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.362559] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA53-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963667250 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44927} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:1176} write_epoch=1, prev=this=0x2ad8dae40ec0 trans_version={val:1736324387223787000, v:0} scn={val:1736324387223787000, v:0} tx_id={txid:42227} prev=(nil) next=(nil) modify_count=0 acc_checksum=500068692 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7190} write_epoch=1, snapshot_version={val:1736326331360782000, v:0}, *this={this=0x2ad8dae40e30 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae40ec0 latest_compact_node=(nil) max_trans_version={val:1736326270722665000, v:0} max_trans_id=44843 max_elr_trans_version={val:1736326270722665000, v:0} max_elr_trans_id=44843 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324387223787000, v:0} min_modify_scn={val:1736324387223787000, v:0}}) [2025-01-08 00:52:11.362667] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA53-0-0] [lt=104][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963667250 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44927} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:1176} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361260435, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361260435, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331360782000, v:0}, tx_id:{txid:44927}, scn:{branch:0, seq:1173}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44927}, tx_desc:{this:0x2ad8b09c3c80, tx_id:{txid:44927}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331362088, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331456574, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:1176}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.362804] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA53-0-0] [lt=135][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44927}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:1176}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.362823] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA53-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44927} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.362879] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA53-0-0] [lt=44][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.362956] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA53-0-0] [lt=71][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: 3F4632FA2D2B0600, int: 1736326331385407}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: 3F4632FA2D2B0600, int: 1736326331385407}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 00E0010000000000, int: 122880}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 00E0010000000000, int: 122880}]}) [2025-01-08 00:52:11.363056] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA53-0-0] [lt=59][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.363850] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=18717696) [2025-01-08 00:52:11.365337] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D5-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326331364729000, start_id=1736326331364729000, end_id=1736326331364729001) [2025-01-08 00:52:11.365625] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA53-0-0] [lt=28] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9818}, to_seq_no={branch:0, seq:1174}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44927} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.376134] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D6-0-0] [lt=33][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae8ff78 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55795} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:2622} write_epoch=1, prev=this=0x2ad8c1563fa8 trans_version={val:1736324386193967000, v:0} scn={val:1736324386193967000, v:0} tx_id={txid:52596} prev=(nil) next=(nil) modify_count=0 acc_checksum=3372950423 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:33080} write_epoch=1, snapshot_version={val:1736326331373312000, v:0}, *this={this=0x2ad8c1563f18 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1563fa8 latest_compact_node=(nil) max_trans_version={val:1736326270179986000, v:0} max_trans_id=55696 max_elr_trans_version={val:1736326270179986000, v:0} max_elr_trans_id=55696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386193967000, v:0} min_modify_scn={val:1736324386193967000, v:0}}) [2025-01-08 00:52:11.376256] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D6-0-0] [lt=120][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae8ff78 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55795} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:2622} write_epoch=1, ctx={this:0x2ad8b78d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361273921, tablet_id:{id:309}, table_iter:0x2ad8b78d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361273921, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331373312000, v:0}, tx_id:{txid:55795}, scn:{branch:0, seq:2619}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55795}, tx_desc:{this:0x2ad8b716cc60, tx_id:{txid:55795}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331375672, commit_ts:-1, finish_ts:-1, timeout_us:29999977, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331473358, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0434d0, mem_ctx:0x2ad89c044a50, tx_scn:{branch:0, seq:2622}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.376351] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D6-0-0] [lt=93][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55795}, data:0x2ad8b78d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:2622}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.376369] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D6-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55795} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.376500] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D6-0-0] [lt=128][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.376513] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D6-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad92a20c6a0, local_datums:0x2ad92a20c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad92a20c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad92a20c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c372, hex: CF9D33FA2D2B0600, int: 1736326331473359}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c37a, hex: CF9D33FA2D2B0600, int: 1736326331473359}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad92a20c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:11.376609] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46492][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D6-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.378663] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D6-0-0] [lt=52] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5991}, to_seq_no={branch:0, seq:2620}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55795} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.387127] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=54] PNIO [ratelimit] time: 1736326331387123, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007582, add_bytes: 0 [2025-01-08 00:52:11.387825] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:11.388408] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA55-0-0] [lt=47][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae90140 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44928} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:9138} write_epoch=1, prev=this=0x2ad963661f38 trans_version={val:1736326270756388001, v:0} scn={val:1736326270756388001, v:0} tx_id={txid:44844} prev=0x2ad963661d80 next=(nil) modify_count=62 acc_checksum=2381987226 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:24555} write_epoch=3, snapshot_version={val:1736326331385863000, v:0}, *this={this=0x2ad8da45dba8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad963661f38 latest_compact_node=0x2ad8dae89d40 max_trans_version={val:1736326270756388001, v:0} max_trans_id=44844 max_elr_trans_version={val:1736326270756388001, v:0} max_elr_trans_id=44844 latest_compact_ts=1736326209717184 last_compact_cnt=59 total_trans_node_cnt=63 max_modify_scn={val:1736326270756388001, v:0} min_modify_scn={val:1736324387226061003, v:0}}) [2025-01-08 00:52:11.388480] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA55-0-0] [lt=71][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae90140 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44928} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:9138} write_epoch=1, ctx={this:0x2ad8b7bcf678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361284045, tablet_id:{id:309}, table_iter:0x2ad8b7bcdb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361284045, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331385863000, v:0}, tx_id:{txid:44928}, scn:{branch:0, seq:3755}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44928}, tx_desc:{this:0x2ad8ba82f180, tx_id:{txid:44928}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331388057, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331479349, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c03f150, mem_ctx:0x2ad89c0406d0, tx_scn:{branch:0, seq:9138}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.388577] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA55-0-0] [lt=95][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44928}, data:0x2ad8b7bcc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:9138}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.388594] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA55-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44928} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.388628] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA55-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.388639] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA55-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96800c6a0, local_datums:0x2ad96800c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96800c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96800c363, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c373, hex: 7AA532FA2D2B0600, int: 1736326331409786}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c37b, hex: 7AA532FA2D2B0600, int: 1736326331409786}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c383, hex: 00F0250000000000, int: 2486272}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c38b, hex: 00F0250000000000, int: 2486272}]}) [2025-01-08 00:52:11.388739] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA55-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.389940] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46492][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA55-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:11443}, to_seq_no={branch:0, seq:3756}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44928} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.390843] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:11.393088] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.403443] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D8-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963667750 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55796} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4079} write_epoch=1, prev=this=0x2ad8c1564380 trans_version={val:1736324386214262000, v:0} scn={val:1736324386214262000, v:0} tx_id={txid:52597} prev=(nil) next=(nil) modify_count=0 acc_checksum=1313669344 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:3643} write_epoch=1, snapshot_version={val:1736326331400474000, v:0}, *this={this=0x2ad8c15642f0 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1564380 latest_compact_node=(nil) max_trans_version={val:1736326270522134001, v:0} max_trans_id=55697 max_elr_trans_version={val:1736326270522134001, v:0} max_elr_trans_id=55697 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386214262000, v:0} min_modify_scn={val:1736324386214262000, v:0}}) [2025-01-08 00:52:11.403552] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D8-0-0] [lt=107][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963667750 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55796} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4079} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361299791, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361299791, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331400474000, v:0}, tx_id:{txid:55796}, scn:{branch:0, seq:4075}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55796}, tx_desc:{this:0x2ad8b09cfec0, tx_id:{txid:55796}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331401792, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331499103, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:4079}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.403654] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D8-0-0] [lt=100][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55796}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4079}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.403673] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D8-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55796} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.403707] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D8-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.403718] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D8-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c372, hex: 600234FA2D2B0600, int: 1736326331499104}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37a, hex: 600234FA2D2B0600, int: 1736326331499104}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:11.403837] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5D8-0-0] [lt=84][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.406289] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46499][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5D8-0-0] [lt=62] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7482}, to_seq_no={branch:0, seq:4076}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55796} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.411450] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24] ====== tenant freeze timer task ====== [2025-01-08 00:52:11.415155] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-01-08 00:52:11.415185] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70704720, tenant_id=1001) [2025-01-08 00:52:11.415270] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=75] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:52:11.417038] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:11.418845] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:11.422298] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:52:11.423366] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58695][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=22] worker thread exit(tenant_->id()=1001, workers_.get_size()=4) [2025-01-08 00:52:11.426655] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5DA-0-0] [lt=12][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963667918 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55797} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4005} write_epoch=1, prev=this=0x2ad8dae8aa70 trans_version={val:1736326270589265000, v:0} scn={val:1736326270589265000, v:0} tx_id={txid:55698} prev=0x2ad8dae8ab40 next=(nil) modify_count=120 acc_checksum=955832032 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:46087} write_epoch=3, snapshot_version={val:1736326331423538000, v:0}, *this={this=0x2ad8c1457a98 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae8aa70 latest_compact_node=0x2ad8dae8ab40 max_trans_version={val:1736326270589265000, v:0} max_trans_id=55698 max_elr_trans_version={val:1736326270589265000, v:0} max_elr_trans_id=55698 latest_compact_ts=1736326270606552 last_compact_cnt=119 total_trans_node_cnt=121 max_modify_scn={val:1736326270589265000, v:0} min_modify_scn={val:1736322641129989001, v:0}}) [2025-01-08 00:52:11.426761] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5DA-0-0] [lt=104][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963667918 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55797} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4005} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361324117, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361324117, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331423538000, v:0}, tx_id:{txid:55797}, scn:{branch:0, seq:4002}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55797}, tx_desc:{this:0x2ad8b09cde60, tx_id:{txid:55797}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331425922, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331523276, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a01d0, mem_ctx:0x2ad89c0a1750, tx_scn:{branch:0, seq:4005}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.426906] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5DA-0-0] [lt=143][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55797}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4005}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.426992] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5DA-0-0] [lt=85][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55797} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.427082] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5DA-0-0] [lt=87][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.427095] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5DA-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: CD6034FA2D2B0600, int: 1736326331523277}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: CD6034FA2D2B0600, int: 1736326331523277}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 821B0A0900000000, int: 151657346}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 821B0A0900000000, int: 151657346}]}) [2025-01-08 00:52:11.427282] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5DA-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.428473] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C63-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=20, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:11.428562] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C63-0-0] [lt=88] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=27739, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:11.428576] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C63-0-0] [lt=12] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=2476, avg_log_submit_to_slide_time=27717) [2025-01-08 00:52:11.428497] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1454, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:11.428649] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=143] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18422, miss_cnt:0, cache_read_size:2253291, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:11.429335] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DA-0-0] [lt=54] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6325}, to_seq_no={branch:0, seq:4003}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55797} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.429880] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA59-0-0] [lt=213][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8da4c49b8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44930} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:6324} write_epoch=1, prev=this=0x2ad8c1566f48 trans_version={val:1736324387335228000, v:0} scn={val:1736324387335228000, v:0} tx_id={txid:42230} prev=(nil) next=(nil) modify_count=0 acc_checksum=1788252403 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:19273} write_epoch=1, snapshot_version={val:1736326331425922000, v:0}, *this={this=0x2ad8c1566eb8 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1566f48 latest_compact_node=(nil) max_trans_version={val:1736326270802951000, v:0} max_trans_id=44846 max_elr_trans_version={val:1736326270802951000, v:0} max_elr_trans_id=44846 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324387335228000, v:0} min_modify_scn={val:1736324387335228000, v:0}}) [2025-01-08 00:52:11.429950] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA59-0-0] [lt=67][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8da4c49b8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44930} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:6324} write_epoch=1, ctx={this:0x2ad8b7855678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361327057, tablet_id:{id:309}, table_iter:0x2ad8b7853b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361327057, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331425922000, v:0}, tx_id:{txid:44930}, scn:{branch:0, seq:4004}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44930}, tx_desc:{this:0x2ad8b94049a0, tx_id:{txid:44930}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331429457, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331523278, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0191d0, mem_ctx:0x2ad89c01a750, tx_scn:{branch:0, seq:6324}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.430094] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA59-0-0] [lt=143][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44930}, data:0x2ad8b7852508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:6324}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.430114] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA59-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44930} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.430139] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA59-0-0] [lt=24][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.430150] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA59-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad955c0c6a0, local_datums:0x2ad955c0c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad955c0c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad955c0c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad955c0c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad955c0c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad955c0c363, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad955c0c373, hex: 004C33FA2D2B0600, int: 1736326331452416}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad955c0c37b, hex: 004C33FA2D2B0600, int: 1736326331452416}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad955c0c383, hex: 0000BA0000000000, int: 12189696}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad955c0c38b, hex: 0000BA0000000000, int: 12189696}]}) [2025-01-08 00:52:11.430314] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA59-0-0] [lt=127][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.430728] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1454) [2025-01-08 00:52:11.434088] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46382][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA59-0-0] [lt=22] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9741}, to_seq_no={branch:0, seq:4005}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44930} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.436681] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:11.436710] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:52:11.442582] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=21497) [2025-01-08 00:52:11.444482] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46382][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5A-0-0] [lt=54] get number(ret=0, service_type_=0, range=1, base_id=1736326331444391000, start_id=1736326331444391000, end_id=1736326331444391001) [2025-01-08 00:52:11.451344] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:11.451466] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=83] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:36, bw:170924, [delay/us]:prepare:14, schedule:126, submit:56, rt:1508, total:1705) [2025-01-08 00:52:11.451482] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:11.451563] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=63] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:36, avg_bw:170929, [delay/us]: prepare:14, schedule:126, submit:56, device:1508, total:1705) [2025-01-08 00:52:11.451579] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:11.451642] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:15, bw:69187, [delay/us]:prepare:15, schedule:133, submit:117, rt:2517, total:2783) [2025-01-08 00:52:11.451654] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:11.451714] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:15, avg_bw:69186, [delay/us]: prepare:15, schedule:133, submit:117, device:2517, total:2783) [2025-01-08 00:52:11.451728] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:11.451750] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:6144, iops:4, bw:24418, [delay/us]:prepare:6, schedule:80, submit:1, rt:2868, total:2957) [2025-01-08 00:52:11.451815] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4295, iops:41, bw:175000, [delay/us]:prepare:7, schedule:247, submit:245, rt:2223, total:2724) [2025-01-08 00:52:11.451828] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:11.451885] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=50] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:6144, avg_iops:4, avg_bw:24420, [delay/us]: prepare:6, schedule:80, submit:1, device:2868, total:2957) [2025-01-08 00:52:11.451903] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4296, avg_iops:41, avg_bw:175010, [delay/us]: prepare:7, schedule:247, submit:245, device:2223, total:2724) [2025-01-08 00:52:11.451913] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:11.451934] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4070, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:259, total:268) [2025-01-08 00:52:11.451958] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:11.452008] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4069, [delay/us]: prepare:0, schedule:0, submit:600000000, device:259, total:268) [2025-01-08 00:52:11.452356] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:11.452533] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=166] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:11.452573] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:11.452584] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:11.452592] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=877kB/s, net_out=368kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:11.455545] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DC-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8da4c4b88 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55798} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:6177} write_epoch=1, prev=this=0x2ad8da45d4a0 trans_version={val:1736324386340473000, v:0} scn={val:1736324386340473000, v:0} tx_id={txid:52599} prev=(nil) next=(nil) modify_count=0 acc_checksum=3388464126 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:5965} write_epoch=1, snapshot_version={val:1736326331448974000, v:0}, *this={this=0x2ad8da45d410 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8da45d4a0 latest_compact_node=(nil) max_trans_version={val:1736326270641542000, v:0} max_trans_id=55699 max_elr_trans_version={val:1736326270641542000, v:0} max_elr_trans_id=55699 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324386340473000, v:0} min_modify_scn={val:1736324386340473000, v:0}}) [2025-01-08 00:52:11.455617] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DC-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8da4c4b88 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55798} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:6177} write_epoch=1, ctx={this:0x2ad8ad751678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361349735, tablet_id:{id:309}, table_iter:0x2ad8ad74fb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361349735, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331448974000, v:0}, tx_id:{txid:55798}, scn:{branch:0, seq:6169}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:55798}, tx_desc:{this:0x2ad8b5e3f480, tx_id:{txid:55798}, state:1, addr:"192.168.159.138:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331454340, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331547292, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0c6150, mem_ctx:0x2ad89c0c76d0, tx_scn:{branch:0, seq:6177}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.455711] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DC-0-0] [lt=93][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:55798}, data:0x2ad8ad74e508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:6177}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.455727] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DC-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55798} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.455757] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DC-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.455769] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DC-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8d3c0c6a0, local_datums:0x2ad8d3c0c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c33c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8d3c0c344, hex: 3139322E3136382E3135392E313338, cstr: 192.168.159.138}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c35b, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8d3c0c363, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c373, hex: C8C434FA2D2B0600, int: 1736326331548872}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c37b, hex: C8C434FA2D2B0600, int: 1736326331548872}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c383, hex: 710B080000000000, int: 527217}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8d3c0c38b, hex: 00D03D0000000000, int: 4050944}]}) [2025-01-08 00:52:11.455855] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DC-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.456094] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA5B-0-0] [lt=33][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963667d70 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44931} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:6176} write_epoch=1, prev=this=0x2ad8c156f8a8 trans_version={val:1736324448500937000, v:0} scn={val:1736324448500937000, v:0} tx_id={txid:42342} prev=0x2ad8c156f6f0 next=(nil) modify_count=2 acc_checksum=3147207185 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:40876} write_epoch=3, snapshot_version={val:1736326331452924000, v:0}, *this={this=0x2ad8c1567298 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=3 list_head=0x2ad8c156f8a8 latest_compact_node=(nil) max_trans_version={val:1736326270858229000, v:0} max_trans_id=44847 max_elr_trans_version={val:1736326270858229000, v:0} max_elr_trans_id=44847 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=3 max_modify_scn={val:1736324448500937000, v:0} min_modify_scn={val:1736324387344938000, v:0}}) [2025-01-08 00:52:11.456271] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA5B-0-0] [lt=175][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963667d70 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44931} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:6176} write_epoch=1, ctx={this:0x2ad8ad0d3678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361352523, tablet_id:{id:309}, table_iter:0x2ad8ad0d1b68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361352523, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331452924000, v:0}, tx_id:{txid:44931}, scn:{branch:0, seq:6170}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44931}, tx_desc:{this:0x2ad8b09dc100, tx_id:{txid:44931}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331454340, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331547294, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c0a88d0, mem_ctx:0x2ad89c0a9e50, tx_scn:{branch:0, seq:6176}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.456352] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA5B-0-0] [lt=80][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44931}, data:0x2ad8ad0d0508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:6176}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.456367] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA5B-0-0] [lt=14][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44931} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.456393] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA5B-0-0] [lt=24][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.456402] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA5B-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96100c6a0, local_datums:0x2ad96100c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c33c, hex: EA03000000000000, int: 1002}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96100c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad96100c363, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c373, hex: D4AE33FA2D2B0600, int: 1736326331477716}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c37b, hex: D4AE33FA2D2B0600, int: 1736326331477716}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c383, hex: 0060870000000000, int: 8871936}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96100c38b, hex: 0060870000000000, int: 8871936}]}) [2025-01-08 00:52:11.456488] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D17DBBA5B-0-0] [lt=47][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.457855] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DC-0-0] [lt=13] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10746}, to_seq_no={branch:0, seq:6170}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55798} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.459123] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.459280] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.459388] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5B-0-0] [lt=37] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:13088}, to_seq_no={branch:0, seq:6171}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44931} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.466293] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46387][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DD-0-0] [lt=16] get number(ret=0, service_type_=0, range=1, base_id=1736326331466087000, start_id=1736326331466087000, end_id=1736326331466087001) [2025-01-08 00:52:11.478640] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068B004-0-0] [lt=75] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.794573879241943359e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=389, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.605895266636067618e+02) [2025-01-08 00:52:11.482009] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5D-0-0] [lt=31][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad8dae90478 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44932} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:85} seq_no={branch:0, seq:7394} write_epoch=1, prev=this=0x2ad8c1458aa0 trans_version={val:1736322641212342000, v:0} scn={val:1736322641212342000, v:0} tx_id={txid:40117} prev=(nil) next=(nil) modify_count=0 acc_checksum=3184665992 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:85} seq_no={branch:0, seq:4058} write_epoch=1, snapshot_version={val:1736326331479512000, v:0}, *this={this=0x2ad8c1458a10 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8c1458aa0 latest_compact_node=(nil) max_trans_version={val:1736326270908340000, v:0} max_trans_id=44849 max_elr_trans_version={val:1736326270908340000, v:0} max_elr_trans_id=44849 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736322641212342000, v:0} min_modify_scn={val:1736322641212342000, v:0}}) [2025-01-08 00:52:11.482151] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5D-0-0] [lt=139][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad8dae90478 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:44932} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:85} seq_no={branch:0, seq:7394} write_epoch=1, ctx={this:0x2ad8b7bcf678, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326361379627, tablet_id:{id:309}, table_iter:0x2ad8b7bcdb68, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326361379627, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326331479512000, v:0}, tx_id:{txid:44932}, scn:{branch:0, seq:3847}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:44932}, tx_desc:{this:0x2ad8ba833240, tx_id:{txid:44932}, state:1, addr:"192.168.159.139:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1736326331480637, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326331574353, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:1}, tx_ctx:0x2ad89c03f150, mem_ctx:0x2ad89c0406d0, tx_scn:{branch:0, seq:7394}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:52:11.482421] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5D-0-0] [lt=267][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:44932}, data:0x2ad8b7bcc508, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:7394}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:52:11.482443] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5D-0-0] [lt=21][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.139", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":3},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:44932} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.482475] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5D-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:52:11.482487] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5D-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad96800c6a0, local_datums:0x2ad96800c6a0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c33c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad96800c344, hex: 3139322E3136382E3135392E313339, cstr: 192.168.159.139}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c353, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c35b, hex: 0300000000000000, int: 3}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad96800c363, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c372, hex: A31E34FA2D2B0600, int: 1736326331506339}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c37a, hex: A31E34FA2D2B0600, int: 1736326331506339}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c382, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad96800c38a, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:52:11.482738] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46498][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5D-0-0] [lt=208][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:52:11.483691] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46380][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DE-0-0] [lt=30][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad9636680a0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:55799} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:3553} write_epoch=1, prev=this=0x2ad8c1570bf0 trans_version={val:1736324508533827000, v:0} scn={val:1736324508533827000, v:0} tx_id={txid:52830} prev=0x2ad8c1570a38 next=(nil) modify_count=4 acc_checksum=2505866000 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326331253224000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:26113} write_epoch=3, snapshot_version={val:1736326331477840000, v:0}, *this={this=0x2ad8c1564d38 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=5 list_head=0x2ad8c1570bf0 latest_compact_node=(nil) max_trans_version={val:1736326270669555000, v:0} max_trans_id=55700 max_elr_trans_version={val:1736326270669555000, v:0} max_elr_trans_id=55700 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=5 max_modify_scn={val:1736324508533827000, v:0} min_modify_scn={val:1736324386362509000, v:0}}) [2025-01-08 00:52:11.483758] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46380][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DE-0-0] [lt=66][errcode=-5024] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:52:11.483921] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46380][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DE-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55799} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.484177] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D17DBBA5D-0-0] [lt=1] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7397}, to_seq_no={branch:0, seq:3848}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:44932} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.486613] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5DE-0-0] [lt=91] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8440}, to_seq_no={branch:0, seq:3550}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55799} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.493311] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.502799] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1736326331501102000, start_id=1736326331501102000, end_id=1736326331501102001) [2025-01-08 00:52:11.509777] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46502][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBD1-0-0] [lt=37][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193836423610, cache_obj->added_lc()=false, cache_obj->get_object_id()=13718, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:11.510427] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58707][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:11.510480] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:52:11.516727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5990) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=50] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000) [2025-01-08 00:52:11.516765] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.516964] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=69] get multi version start(ls_id={id:1}, tablet_id={id:60379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517048] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517189] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517332] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517534] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517605] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60429}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=77] get multi version start(ls_id={id:1}, tablet_id={id:60507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517862] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.517993] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.518054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.518112] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.518446] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.518586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:101011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.518688] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846994}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.518754] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.518818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50124}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.518902] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.518970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.519031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.519168] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.519304] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=78] get multi version start(ls_id={id:1}, tablet_id={id:50355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.519367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.519567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.519632] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.519694] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101098}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.519818] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=64] get multi version start(ls_id={id:1}, tablet_id={id:60387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.519869] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=65] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.519948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=56] get multi version start(ls_id={id:1}, tablet_id={id:419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.520015] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.520076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.520138] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.520274] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.520392] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101036}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.520568] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=119] get multi version start(ls_id={id:1}, tablet_id={id:60005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.521345] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=155] get multi version start(ls_id={id:1}, tablet_id={id:60130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.521339] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E0-0-0] [lt=36][errcode=-5024] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:52:11.521398] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E0-0-0] [lt=0][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55800} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.521434] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:101064}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.521664] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60129}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.521735] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.521965] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=170] get multi version start(ls_id={id:1}, tablet_id={id:50393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522056] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101080}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522120] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522183] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:508}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522401] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=162] get multi version start(ls_id={id:1}, tablet_id={id:197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522465] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522647] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522708] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522769] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=105] get multi version start(ls_id={id:1}, tablet_id={id:101083}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.522995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=30] get multi version start(ls_id={id:1}, tablet_id={id:101062}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523145] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:129}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523400] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523461] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523588] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523658] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101018}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523721] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:101108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.523976] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.524060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.524130] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101034}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.524363] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.524550] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.524825] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.525032] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101084}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.525097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.525304] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=66] get multi version start(ls_id={id:1}, tablet_id={id:60233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.525370] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50219}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.525412] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:11.525432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.525493] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.525554] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.525614] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.525672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101068}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.526070] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.529103] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.529187] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:60140}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.529552] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.529633] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50484}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.529710] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.529770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.529832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.529939] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530152] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530291] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530370] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530480] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530571] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530636] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530755] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101073}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530812] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530870] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530925] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:128}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.530983] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531040] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531097] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101099}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531281] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101092}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531845] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60484}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531904] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.531963] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101031}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532079] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=273] get multi version start(ls_id={id:1}, tablet_id={id:60459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532471] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846983}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532646] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532805] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846990}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.532958] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533086] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533207] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60352}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533388] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBD4-0-0] [lt=0][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193836447220, cache_obj->added_lc()=false, cache_obj->get_object_id()=13719, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:11.533422] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533548] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533587] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E0-0-0] [lt=23] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:21720}, to_seq_no={branch:0, seq:3897}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55800} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.533609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533670] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533848] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50140}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.533968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101067}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534027] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:429}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534179] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534306] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534368] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534466] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534602] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60309}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101027}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846986}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534850] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.534941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.535039] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.535286] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.535360] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.535642] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=211] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846977}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.535732] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.535797] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.535856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.535918] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.536306] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.536510] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.536578] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.536696] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.536766] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.536829] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101082}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.537043] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.537113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.537260] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.537329] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.537758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.537829] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.537960] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.538089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.539629] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:52:11.540019] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=49] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:444K/98:210938 disk:444K/98:210938 t1m2:184K/39:142420 t1m2g0:184K/39:142420 t1002m2:192K/44:289942 t1002m2g0:192K/44:289942 t1001m2:68K/15:157339 t1001m2g0:68K/15:157339 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:11.542090] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.543843] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=227] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6726971392, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:11.543907] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.544430] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=6258] get multi version start(ls_id={id:1}, tablet_id={id:50239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.544438] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2253413}], scn_array=[cnt:1, {val:1736326331356286001, v:0}], curr_size=122, accum_size=122, time_cost=10396) [2025-01-08 00:52:11.544612] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:297}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.544715] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:60517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.545038] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=104] get multi version start(ls_id={id:1}, tablet_id={id:60478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.545142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:50420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.545332] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.545439] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=52] get multi version start(ls_id={id:1}, tablet_id={id:60248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.546429] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.546619] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60502}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.546931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:200009}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.547916] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=522] get multi version start(ls_id={id:1}, tablet_id={id:410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.548028] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:50507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.548950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=854] get multi version start(ls_id={id:1}, tablet_id={id:60147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.549300] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=217] get multi version start(ls_id={id:1}, tablet_id={id:60361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.549403] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:60116}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.549669] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=205] get multi version start(ls_id={id:1}, tablet_id={id:60323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.549770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:101057}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.549927] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.550081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=94] get multi version start(ls_id={id:1}, tablet_id={id:60158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.550271] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=61] get multi version start(ls_id={id:1}, tablet_id={id:60358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.550373] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.550468] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.550561] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.550654] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.550756] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:60381}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.550850] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.551019] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.551294] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.551427] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=66] get multi version start(ls_id={id:1}, tablet_id={id:101033}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.551528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.551626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.551736] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:60251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.551832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.552554] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=138] get multi version start(ls_id={id:1}, tablet_id={id:50392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.553239] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=539] get multi version start(ls_id={id:1}, tablet_id={id:485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.553351] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:60181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.553449] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101066}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.553544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.553638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.553731] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.553825] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:320}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.553995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846980}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.554324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=125] get multi version start(ls_id={id:1}, tablet_id={id:50401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.554466] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=55] get multi version start(ls_id={id:1}, tablet_id={id:499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.554565] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.554655] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.554743] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.554831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.554917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.555005] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.555089] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:101030}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.555265] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:101094}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.555361] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101020}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.555449] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.555537] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50117}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.556609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=1011] get multi version start(ls_id={id:1}, tablet_id={id:101044}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.556669] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:100002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.557350] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.557412] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.557470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.557527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.557585] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50440}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.557643] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.557700] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:293}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.557911] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50395}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.557972] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:118}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558130] INFO [LIB] stat (utility.h:1210) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E1-0-0] [lt=35] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=39, stat_interval=1000000, avg (count/cost)=91, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:11.558143] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558182] INFO [LIB] stat (utility.h:1210) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E1-0-0] [lt=21] [PALF STAT APPEND COST TIME](cur_stat_count=40, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:11.558375] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558504] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558568] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558691] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:60450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558781] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:445}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558840] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101017}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.558898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559006] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=57] get multi version start(ls_id={id:1}, tablet_id={id:101013}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559494] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559769] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559833] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559892] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60256}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.559947] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101010}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101072}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560183] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560307] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=70] get multi version start(ls_id={id:1}, tablet_id={id:50431}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560387] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101088}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560502] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101069}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560560] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560617] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560732] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60486}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560789] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560875] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60493}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560938] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50410}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.560996] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561015] INFO [SERVER] refresh_avail_memory (ob_table_load_assigned_memory_manager.cpp:117) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=91] ObTableLoadAssignedMemoryManager::refresh_avail_memory(MTL_ID()=1, avail_memory=107374182, chunk_count_=0, avail_sort_memory_=107374182, avail_memory_=107374182) [2025-01-08 00:52:11.561054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561348] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=30] get multi version start(ls_id={id:1}, tablet_id={id:191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561489] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60108}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561546] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561604] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50128}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561662] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101060}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561718] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50256}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561776] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561834] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101100}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:279}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.561950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.562008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.562065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.562122] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.562264] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.562411] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101022}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.562529] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=64] get multi version start(ls_id={id:1}, tablet_id={id:151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.562641] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.562706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.562906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=80] get multi version start(ls_id={id:1}, tablet_id={id:60515}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563087] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=71] get multi version start(ls_id={id:1}, tablet_id={id:482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563170] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563291] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=67] get multi version start(ls_id={id:1}, tablet_id={id:50471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563464] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50226}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563644] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563698] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563756] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:305}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563814] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.563986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564044] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564102] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564158] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564337] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101035}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564392] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:5}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564479] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564602] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50145}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564660] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101058}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564774] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50485}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101081}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.564943] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:245}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565001] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101086}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101052}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565223] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565289] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565347] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101019}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565478] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565543] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101079}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565602] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565660] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565773] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565831] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565888] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.565946] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566003] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50238}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566114] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:1}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566172] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566230] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566360] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:50377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566419] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566477] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60320}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566617] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566674] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566731] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200010}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.566788] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.567780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.567882] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.567974] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.568065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.568161] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:50436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.568336] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=118] get multi version start(ls_id={id:1}, tablet_id={id:50160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.568425] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.568535] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60393}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.568682] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50451}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.568815] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=58] get multi version start(ls_id={id:1}, tablet_id={id:50366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.568916] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.569008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.569175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.569403] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=62] [PALF STAT WRITE LOG COST TIME](cur_stat_count=39, stat_interval=1000000, avg (count/cost)=1914, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:11.569353] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=117] get multi version start(ls_id={id:1}, tablet_id={id:200006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.569569] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=39, stat_interval=1000000, avg (count/cost)=15, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:11.569610] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT LOG LIFE TIME](cur_stat_count=39, stat_interval=1000000, avg (count/cost)=4451, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:11.569682] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=204] get multi version start(ls_id={id:1}, tablet_id={id:60472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.569737] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=37] rpc statistics(total_trans_msg_count=62, total_batch_msg_count=0) [2025-01-08 00:52:11.569986] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570171] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570332] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=103] get multi version start(ls_id={id:1}, tablet_id={id:348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570425] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570523] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:50438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570613] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570817] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60401}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.570996] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.571086] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.571257] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.571392] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=60] get multi version start(ls_id={id:1}, tablet_id={id:60146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.571499] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=52] get multi version start(ls_id={id:1}, tablet_id={id:319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.571590] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.571770] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=106] get multi version start(ls_id={id:1}, tablet_id={id:385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.571865] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.571954] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572041] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572129] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572288] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572382] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572477] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:50452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572566] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:155}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572653] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50309}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572738] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572855] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50419}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.572949] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.573544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=534] get multi version start(ls_id={id:1}, tablet_id={id:60470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.573639] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.573736] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:453}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.573918] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.574011] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.574099] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.574276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=120] get multi version start(ls_id={id:1}, tablet_id={id:60525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.574596] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50125}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.574663] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.574633] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58707][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6AB9-0-0] [lt=176] init point thread id with(&point=0x5572d50e7e00, idx_=1363, point=[thread id=58707, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58707) [2025-01-08 00:52:11.574723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846979}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.574780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.574837] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60344}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.574950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50157}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575101] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575168] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575286] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575261] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46388][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E2-0-0] [lt=6][errcode=-5024] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:52:11.575313] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46388][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E2-0-0] [lt=0][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant backup data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55801} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.575348] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575462] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575578] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575636] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575693] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60405}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575750] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60150}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575808] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.575865] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576001] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576064] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576120] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101043}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576178] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576295] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576357] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=75] get multi version start(ls_id={id:1}, tablet_id={id:50225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576672] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576733] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:460}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576791] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:167}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576849] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576906] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:149}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.576989] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50482}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.577050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.577336] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.577333] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E2-0-0] [lt=13] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7077}, to_seq_no={branch:0, seq:3664}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55801} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.577399] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.577456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.577745] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.577804] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.577991] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=115] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846988}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578054] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578369] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101032}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578436] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101089}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578495] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578550] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:208}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578608] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200008}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578681] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578744] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.578858] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:124}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.579075] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.579457] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=310] get multi version start(ls_id={id:1}, tablet_id={id:60455}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.579745] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.579836] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:200005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.579899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:465}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.579959] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.581229] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.581452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:50332}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.581563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:60128}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.581638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50472}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.581854] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50474}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.581934] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.581995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582058] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50432}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582396] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=170] get multi version start(ls_id={id:1}, tablet_id={id:60168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60392}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582533] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50296}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582661] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582724] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:478}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50294}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.582966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:140}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.583142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=101] get multi version start(ls_id={id:1}, tablet_id={id:200001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.583856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:101028}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.583941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584006] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584069] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50253}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584507] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584611] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:60378}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584703] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:507}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584766] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584864] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50155}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584928] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101021}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.584988] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585049] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585109] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585344] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=102] get multi version start(ls_id={id:1}, tablet_id={id:50487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:60292}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585575] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:60196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585638] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50350}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585740] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60239}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585874] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.585967] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586092] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50111}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586154] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586539] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=155] get multi version start(ls_id={id:1}, tablet_id={id:209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586617] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586684] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586748] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586810] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586885] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.586948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50116}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60528}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587071] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587194] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101037}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587324] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=72] get multi version start(ls_id={id:1}, tablet_id={id:390}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:263}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587530] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587619] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587678] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:101054}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50444}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.587940] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588032] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50165}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60395}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588188] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=123] get multi version start(ls_id={id:1}, tablet_id={id:50119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588461] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588525] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588640] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50123}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588738] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101041}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588840] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:525}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.588971] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:200012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589035] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50369}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589096] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589157] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101097}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589435] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589501] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:50516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589563] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589624] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101024}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589685] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589746] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:236}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589899] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.589972] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:60225}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:60131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590129] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590289] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590381] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60423}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590482] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:483}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=86] get multi version start(ls_id={id:1}, tablet_id={id:50439}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590718] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60391}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590814] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101055}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590873] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50496}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.590972] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591042] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60357}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591151] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=54] get multi version start(ls_id={id:1}, tablet_id={id:113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591286] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591392] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:60458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:123}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591508] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591597] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50290}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591686] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60437}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591743] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591830] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591891] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:454}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.591948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:471}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592031] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:450}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592200] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101063}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=94] get multi version start(ls_id={id:1}, tablet_id={id:50376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592525] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:375}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592610] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:3}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592711] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592812] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592903] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:219}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.592961] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.593316] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=292] get multi version start(ls_id={id:1}, tablet_id={id:168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.593424] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846985}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.593484] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:141}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.593545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.593607] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.593668] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101040}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.593822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.593937] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846978}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.593950] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.594004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50491}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.594125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50494}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.594343] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101016}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.594527] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=87] get multi version start(ls_id={id:1}, tablet_id={id:248}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.594637] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=26] get multi version start(ls_id={id:1}, tablet_id={id:60153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.594742] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:50386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.594893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:101047}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.595010] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.595155] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=61] get multi version start(ls_id={id:1}, tablet_id={id:50265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.595330] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.595487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:60307}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.595593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.595692] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:50411}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.595840] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:60505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.595987] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=60] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846993}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.596124] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:50459}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.596544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.596677] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.596785] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:504}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.596887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=23] get multi version start(ls_id={id:1}, tablet_id={id:101059}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.596984] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:50131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:50262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597295] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597359] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:324}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597419] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:441}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597489] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597573] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60219}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597635] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:130}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597702] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:470}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597821] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:262}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.597927] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=25] get multi version start(ls_id={id:1}, tablet_id={id:50513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598029] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=24] get multi version start(ls_id={id:1}, tablet_id={id:60155}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598107] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598165] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598291] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598352] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598410] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598468] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598526] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598616] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:380}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598717] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598808] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598868] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50345}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.598927] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:101107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.599017] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50166}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.599108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.599169] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.599541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=315] get multi version start(ls_id={id:1}, tablet_id={id:295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.599631] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.599747] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.599812] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.599871] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60143}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.599983] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=58] get multi version start(ls_id={id:1}, tablet_id={id:60406}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600040] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600098] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600156] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600308] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600409] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101046}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600514] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=49] get multi version start(ls_id={id:1}, tablet_id={id:60427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600576] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60265}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600634] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:430}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50464}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600838] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60252}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600929] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.600998] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:516}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:117}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601140] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50295}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601291] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601389] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601449] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50152}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601546] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60475}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601635] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:251}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601723] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601784] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601842] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601895] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50291}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.601991] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:101045}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602077] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50371}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602139] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:497}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602275] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:502}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602371] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:488}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602431] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602526] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602581] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60453}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:131}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602753] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60442}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602808] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602902] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:50243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.602989] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60316}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.603045] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.603125] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:101015}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.603272] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:246}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.603358] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60499}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.603458] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:101078}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.603555] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.603617] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:148}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.603872] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.603968] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.604032] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101049}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.604108] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.604175] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:365}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.605446] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=1198] get multi version start(ls_id={id:1}, tablet_id={id:220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.605520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:50151}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.605727] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.605919] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101101}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.606163] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.606317] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=92] get multi version start(ls_id={id:1}, tablet_id={id:366}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.606485] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=111] get multi version start(ls_id={id:1}, tablet_id={id:50255}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.606560] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46501][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E3-0-0] [lt=216] get number(ret=0, service_type_=0, range=1, base_id=1736326331606107000, start_id=1736326331606107000, end_id=1736326331606107001) [2025-01-08 00:52:11.606741] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846995}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.606839] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607006] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60135}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607071] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:169}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50404}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607190] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:50202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607364] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=104] get multi version start(ls_id={id:1}, tablet_id={id:213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607589] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=168] get multi version start(ls_id={id:1}, tablet_id={id:514}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:256}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607712] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:119}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607832] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50412}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607892] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101091}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.607951] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101008}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608011] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50426}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608072] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608133] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:356}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608257] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101039}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608327] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101112}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101029}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60385}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608507] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608626] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608683] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608862] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60109}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608927] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.608989] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609045] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60325}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609145] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60524}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609287] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=83] get multi version start(ls_id={id:1}, tablet_id={id:50397}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609378] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:433}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609478] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60374}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609542] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609609] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:60206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609669] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:333}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609730] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609898] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=112] get multi version start(ls_id={id:1}, tablet_id={id:50206}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.609961] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610023] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:424}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610293] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610369] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:100004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610439] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610504] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610567] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60500}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610670] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60389}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610775] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:50422}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610877] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50129}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.610983] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.611129] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:101095}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.611258] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101070}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.611351] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.611450] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.611551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.611651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60249}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.611750] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.611966] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=156] get multi version start(ls_id={id:1}, tablet_id={id:101075}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.612058] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:50182}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.612148] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:60373}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.612296] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=89] get multi version start(ls_id={id:1}, tablet_id={id:60197}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.612393] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846987}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.612457] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.612720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101077}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.612787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:328}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.612881] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:142}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=58] get multi version start(ls_id={id:1}, tablet_id={id:60118}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613268] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:101071}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=50] get multi version start(ls_id={id:1}, tablet_id={id:190}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613450] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:101093}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613553] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101012}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613651] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:420}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613714] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50220}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613812] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:360}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.613911] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:364}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.614008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:101065}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.614066] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.614119] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:309}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.614338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:438}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.614441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50495}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.614504] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:214}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.614718] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=143] get multi version start(ls_id={id:1}, tablet_id={id:394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.614844] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=51] get multi version start(ls_id={id:1}, tablet_id={id:60462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.614949] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:329}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.615018] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50462}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.615149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:492}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.618109] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:116}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.618354] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=175] get multi version start(ls_id={id:1}, tablet_id={id:60372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.618476] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:100003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.618549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.619079] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=471] get multi version start(ls_id={id:1}, tablet_id={id:105}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.619374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:331}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.619467] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:342}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.619532] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50435}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.619635] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101014}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.619738] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60468}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.619823] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:200011}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.619886] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60308}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.619948] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:334}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.620009] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.621173] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=225] get multi version start(ls_id={id:1}, tablet_id={id:50506}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.621645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=408] get multi version start(ls_id={id:1}, tablet_id={id:50461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.621750] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=45] get multi version start(ls_id={id:1}, tablet_id={id:261}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.621853] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:147}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.621917] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101102}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.621979] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50103}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50505}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622143] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:358}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622303] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=100] get multi version start(ls_id={id:1}, tablet_id={id:101004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622407] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622557] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:457}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622657] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60394}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622856] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.622974] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.623074] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60319}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.623322] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:458}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.623420] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:60326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.623483] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:379}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.623544] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50326}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.623606] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.623662] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60467}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.623758] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:60283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.623822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.624306] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=342] get multi version start(ls_id={id:1}, tablet_id={id:413}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.624425] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=52] get multi version start(ls_id={id:1}, tablet_id={id:50235}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.624495] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:240}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.624593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:50445}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.624698] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=47] get multi version start(ls_id={id:1}, tablet_id={id:50503}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.624762] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50213}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.624822] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:200007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.624908] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:60463}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.624970] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846981}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.625122] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:133}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.625286] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101050}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.625350] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60362}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.625447] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50407}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.625545] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:409}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.626127] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:100001}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.626285] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50284}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.626350] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:243}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.626449] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60498}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.626537] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50327}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.626598] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60124}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.626780] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50429}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.626879] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60436}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.627063] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50359}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.627449] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=27] get multi version start(ls_id={id:1}, tablet_id={id:50453}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.627556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:60223}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.627625] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:101056}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.627685] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50210}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.627807] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:386}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.627910] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60511}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628012] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50383}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628076] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:434}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628137] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60400}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628307] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50244}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628385] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:180}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628483] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628551] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:399}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628615] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60207}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628777] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50154}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628876] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101023}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.628941] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50317}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.629078] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=76] get multi version start(ls_id={id:1}, tablet_id={id:146}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.629178] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:101007}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.629313] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=79] get multi version start(ls_id={id:1}, tablet_id={id:101048}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.629372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:330}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.629463] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=36] get multi version start(ls_id={id:1}, tablet_id={id:60445}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.629549] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:101026}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.629629] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50372}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.629720] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=35] get multi version start(ls_id={id:1}, tablet_id={id:50260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.629887] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=68] get multi version start(ls_id={id:1}, tablet_id={id:50141}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.630020] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=28] get multi version start(ls_id={id:1}, tablet_id={id:60519}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.630340] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=60] get multi version start(ls_id={id:1}, tablet_id={id:50402}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.630487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:60189}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.630560] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:50211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.630671] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50354}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.630771] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50215}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.630846] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=22] get multi version start(ls_id={id:1}, tablet_id={id:50348}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.630907] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.630967] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.631084] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:513}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.631173] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.631391] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=142] get multi version start(ls_id={id:1}, tablet_id={id:428}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.631586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=74] get multi version start(ls_id={id:1}, tablet_id={id:60523}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.631687] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:50456}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.631755] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.631860] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:50106}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.631963] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:101025}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.632065] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:60188}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.632166] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.632428] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=200] get multi version start(ls_id={id:1}, tablet_id={id:60141}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.632528] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:50196}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.632586] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:60132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.632680] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:104}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.632778] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:487}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.632874] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:4}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.632937] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633000] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=19] get multi version start(ls_id={id:1}, tablet_id={id:100005}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:355}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633147] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60224}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633270] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:522}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633374] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846996}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633432] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:351}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633520] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60120}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633578] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60377}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633760] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:60512}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.633928] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=97] get multi version start(ls_id={id:1}, tablet_id={id:160}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=46] get multi version start(ls_id={id:1}, tablet_id={id:50466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634131] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=37] get multi version start(ls_id={id:1}, tablet_id={id:50398}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634276] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60115}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634383] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=48] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846991}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634443] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:200002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634498] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50259}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634583] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=31] get multi version start(ls_id={id:1}, tablet_id={id:60242}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634645] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:200004}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634700] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634787] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=33] get multi version start(ls_id={id:1}, tablet_id={id:50254}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.634973] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=129] get multi version start(ls_id={id:1}, tablet_id={id:50370}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635036] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60117}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635090] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:260}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635187] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101085}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635342] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=98] get multi version start(ls_id={id:1}, tablet_id={id:387}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635441] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:60209}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635529] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50518}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635585] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50454}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635680] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:466}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635769] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:50335}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635830] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:361}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635926] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.635995] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:60388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.636057] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:306}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.636142] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:60443}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.637033] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C67-0-0] [lt=13][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=18514, log_task={header:{begin_lsn:{lsn:2253413}, end_lsn:{lsn:2253535}, log_id:18514, min_scn:{val:1736326331356286001, v:0}, max_scn:{val:1736326331356286001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2253291}, prev_proposal_id:1, committed_end_lsn:{lsn:2253413}, data_checksum:940009244, accum_checksum:178358887, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326331528728, freeze_ts:1736326331528728, submit_ts:1736326331528737, flushed_ts:1736326331545377, gen_to_freeze cost time:0, gen_to_submit cost time:9, submit_to_flush cost time:16640}, fs_cb_begin_ts=1736326331637020, log_life_time=108292) [2025-01-08 00:52:11.638080] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:132}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638179] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=21] get multi version start(ls_id={id:1}, tablet_id={id:101087}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638314] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60123}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638382] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:50153}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638446] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60403}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638509] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:101003}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638632] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50483}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638803] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60490}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638873] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:336}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638935] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50233}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.638996] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50427}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.639055] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.639115] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60315}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.639238] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:484}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.639308] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50191}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.639367] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:343}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.639452] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:50501}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.639514] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:60232}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.639576] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101051}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.639931] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846982}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640004] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:100006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640061] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50473}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640117] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:60480}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640172] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=11] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846984}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640372] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=112] get multi version start(ls_id={id:1}, tablet_id={id:461}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640474] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50158}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640541] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=18] get multi version start(ls_id={id:1}, tablet_id={id:60382}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640604] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50477}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640666] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:60396}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640728] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60237}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640825] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=40] get multi version start(ls_id={id:1}, tablet_id={id:101061}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640893] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:60202}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.640950] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:50181}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641050] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:60211}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641113] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50241}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641174] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641360] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:101076}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641487] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=62] get multi version start(ls_id={id:1}, tablet_id={id:50168}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641593] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=44] get multi version start(ls_id={id:1}, tablet_id={id:416}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641660] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60454}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641757] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:121}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641860] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:50107}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.641920] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=13] get multi version start(ls_id={id:1}, tablet_id={id:50363}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642081] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:101006}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642149] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=17] get multi version start(ls_id={id:1}, tablet_id={id:60272}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642304] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=20] get multi version start(ls_id={id:1}, tablet_id={id:250}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642405] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=43] get multi version start(ls_id={id:1}, tablet_id={id:452}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642470] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:50222}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642558] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60446}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642618] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=12] get multi version start(ls_id={id:1}, tablet_id={id:60114}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642706] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=32] get multi version start(ls_id={id:1}, tablet_id={id:50283}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642794] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:425}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642890] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] get multi version start(ls_id={id:1}, tablet_id={id:421}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.642987] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:154}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1736322632406836001}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643086] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50388}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643236] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=92] get multi version start(ls_id={id:1}, tablet_id={id:60221}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643337] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50520}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643437] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=42] get multi version start(ls_id={id:1}, tablet_id={id:50247}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643577] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=39] get multi version start(ls_id={id:1}, tablet_id={id:1152921504606846989}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643671] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=38] get multi version start(ls_id={id:1}, tablet_id={id:101074}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643734] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:469}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643795] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=14] get multi version start(ls_id={id:1}, tablet_id={id:60192}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643883] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=34] get multi version start(ls_id={id:1}, tablet_id={id:60376}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.643945] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=15] get multi version start(ls_id={id:1}, tablet_id={id:50323}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.644161] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=133] get multi version start(ls_id={id:1}, tablet_id={id:479}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.644338] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=110] get multi version start(ls_id={id:1}, tablet_id={id:101002}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.644456] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=16] get multi version start(ls_id={id:1}, tablet_id={id:101053}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.644556] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=41] REACH SYSLOG RATE LIMIT [frequency] [2025-01-08 00:52:11.648918] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284BCD-0-0] [lt=34] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:11.651625] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6000) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=0] get multi version start(ls_id={id:1}, tablet_id={id:101110}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1736324526571484000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1736324515839366000, max_merged_snapshot=1) [2025-01-08 00:52:11.651982] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:116) [46306][T1_TimerWK2][T1][YB42C0A89F89-00062B2D17385728-0-0] [lt=0] SSTableGCTask(cost_ts=135839) [2025-01-08 00:52:11.656844] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46379][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBE24-0-0] [lt=118][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193836570677, cache_obj->added_lc()=false, cache_obj->get_object_id()=13720, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:11.658403] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=26148) [2025-01-08 00:52:11.660202] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2232185}], scn_array=[cnt:1, {val:1736326331364729002, v:0}], curr_size=122, accum_size=488, time_cost=12150) [2025-01-08 00:52:11.660620] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8881, offset=2523975}, end_cursor:ObLogCursor{file_id=1, log_id=8882, offset=2524280}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894dbc050, len:3257, log_data_len:305, seq:8881, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:11.661376] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E83-0-0] [lt=28] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:11.661438] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E83-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326331364729001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326331364729001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:11.661554] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=40, stat_interval=1000000, avg (count/cost)=726, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:11.661689] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=40, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:11.669477] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E83-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9382) [2025-01-08 00:52:11.674635] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5E6-0-0] [lt=15][errcode=-5024] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:52:11.674700] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5E6-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55803} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.681268] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E6-0-0] [lt=34] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:14386}, to_seq_no={branch:0, seq:7123}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55803} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.686501] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=162] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8882, offset=2524280}, end_cursor:ObLogCursor{file_id=1, log_id=8883, offset=2524577}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894dc0050, len:2952, log_data_len:297, seq:8882, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:11.687282] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E83-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:11.687334] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E83-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326331364729002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326331364729002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:11.694129] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.700156] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=128] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8883, offset=2524577}, end_cursor:ObLogCursor{file_id=1, log_id=8884, offset=2524874}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894dc4050, len:2655, log_data_len:297, seq:8883, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:11.701741] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=94] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8884, offset=2524874}, end_cursor:ObLogCursor{file_id=1, log_id=8885, offset=2525171}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894dc8050, len:2358, log_data_len:297, seq:8884, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:11.704097] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5E8-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55804} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.709638] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5E8-0-0] [lt=64] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10793}, to_seq_no={branch:0, seq:3597}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55804} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.712184] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1736326331712181, bytes: 889066264, bw: 1.009299 MB/s, add_ts: 1005656, add_bytes: 1064313 [2025-01-08 00:52:11.726924] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2266253}, scn={val:1736326331388057004, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:52:11.727304] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324525712447000) [2025-01-08 00:52:11.727362] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=56] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2266253}, scn_:{val:1736326331388057004, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326331823619, first_handle_ts_:1736326331727154, replay_cost_:205, retry_cost_:-1, read_log_buf_:0x2ad964204160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326331726695, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2266309}, next_to_submit_scn_:{val:1736326331388057005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2266197}, end_lsn:{lsn:2266309}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2266309}, readable_log_tail:{lsn:2266309}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64838459, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326331388057004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326331388057004, v:0}, data_checksum:2477830904, flag:2393}}, init_mode_version:0, accumulate_checksum:3431830613, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326331388057004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18526, miss_cnt:0, cache_read_size:2266309, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=3, ret=0) [2025-01-08 00:52:11.736790] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46384][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5EA-0-0] [lt=32][errcode=-5024] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:52:11.736864] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46384][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5EA-0-0] [lt=0][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55805} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.738269] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5EA-0-0] [lt=61] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7124}, to_seq_no={branch:0, seq:3798}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55805} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.741364] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:52:11.744535] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:11.744578] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D59-0-0] [lt=89] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:52:11.758561] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5EC-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55806} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.759721] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5EC-0-0] [lt=55] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5328}, to_seq_no={branch:0, seq:2962}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55806} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.770741] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5ED-0-0] [lt=54] get number(ret=0, service_type_=0, range=1, base_id=1736326331768371000, start_id=1736326331768371000, end_id=1736326331768371001) [2025-01-08 00:52:11.780716] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5EE-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55807} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.782465] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5EE-0-0] [lt=64] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5036}, to_seq_no={branch:0, seq:3684}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55807} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.794266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.800780] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5F0-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55808} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.802068] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5F0-0-0] [lt=55] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7089}, to_seq_no={branch:0, seq:4917}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55808} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.809540] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50507][T1002_L0_G0][T1002][YB42C0A89F8B-00062B2D1B5BBABA-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1736326331807027000, start_id=1736326331807027000, end_id=1736326331807027001) [2025-01-08 00:52:11.824494] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB62C-0-0] [lt=361] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:11.824548] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB62C-0-0] [lt=38] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326331356286003, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326331356286003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:11.832782] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2257875, this={dio_aligned_buf:{buf_write_offset:1101, buf_padding_size:2995, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:24687, truncate_used_ts:5578}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2256896, buf_len=122, total_write_size=12050, total_write_size_after_dio=417792, ob_pwrite_used_ts=296064, count=99, each_pwrite_cost=2990) [2025-01-08 00:52:11.835350] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5F4-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55810} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.836703] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5F4-0-0] [lt=43] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5057}, to_seq_no={branch:0, seq:2439}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55810} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.844122] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:52:11.844162] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=28] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=209) [2025-01-08 00:52:11.844297] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=24] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:52:11.844361] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=55] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:52:11.844375] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=186) [2025-01-08 00:52:11.844556] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=45] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:11.844650] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=44] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:52:11.845987] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46505][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5F5-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326331845254000, start_id=1736326331845254000, end_id=1736326331845254001) [2025-01-08 00:52:11.854340] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58708][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:11.854415] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58708][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5E2C-0-0] [lt=45] init point thread id with(&point=0x5572d50e7e40, idx_=1364, point=[thread id=58708, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58708) [2025-01-08 00:52:11.854713] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-01-08 00:52:11.856336] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=426] gc stale ls task succ [2025-01-08 00:52:11.856780] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326331657794000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=198974, in_cluster_service=true, cluster_version={val:1736326331470929000, v:0}, min_cluster_version={val:1736326331470929000, v:0}, max_cluster_version={val:1736326331470929000, v:0}, get_cluster_version_err=0, cluster_version_delta=385839, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=37004, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=586353755, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:52:11.859458] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5F6-0-0] [lt=0][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55811} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.862954] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5F6-0-0] [lt=26] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9781}, to_seq_no={branch:0, seq:7491}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55811} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.863537] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=42] [APPLY STAT CB APPEND COST TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=140, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:52:11.863565] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=28] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=4654, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:52:11.863575] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:52:11.863582] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=-685, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:52:11.863589] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=4110, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:52:11.865263] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=18717696) [2025-01-08 00:52:11.866154] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58709][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:11.868878] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=581, get_gts_with_stc_cnt=71, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=71, try_wait_gts_elapse_cnt=0) [2025-01-08 00:52:11.868921] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=41] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:52:11.868949] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=518, get_gts_with_stc_cnt=70, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=71, try_wait_gts_elapse_cnt=0) [2025-01-08 00:52:11.869667] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:11.870498] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-01-08 00:52:11.878030] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46379][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BEDC-0-0] [lt=47] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:52:11.883164] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5345249713, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:11.883331] INFO [LIB] do_thread_recycle (threads.cpp:163) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=96] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:52:11.883346] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5345249713, tenant_id:500}, cur_thread_count=3, cnt=-1, new_thread_count=2) [2025-01-08 00:52:11.884698] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC458-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:11.888083] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:11.892954] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE9-0-0] [lt=53] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:52:11.893000] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984CE9-0-0] [lt=42] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:52:11.893395] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:11.894536] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=113][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.897361] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.897426] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.897475] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.897516] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=50] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:11.899293] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=24] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:11.899420] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=124] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:52:11.899421] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8499A-0-0] [lt=26] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:11.899450] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46156][SerScheQueue0][T0][YB42C0A89F89-00062B2D19A8499A-0-0] [lt=25] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:52:11.900022] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46493][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5F8-0-0] [lt=9][errcode=-5024] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:52:11.900104] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46493][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5F8-0-0] [lt=0][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55812} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.901762] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5F8-0-0] [lt=40] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:11517}, to_seq_no={branch:0, seq:5358}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55812} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.917888] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:11.920806] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=9] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:11.921593] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=85] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:11.922144] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=108] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:11.922299] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=82] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326324430078000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:11.922358] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=56] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326324430078000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326324430078000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:52:11.925856] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58709][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5E2F-0-0] [lt=27] init point thread id with(&point=0x5572d50e7e80, idx_=1365, point=[thread id=58709, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58709) [2025-01-08 00:52:11.928445] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58710][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:11.929131] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread created(tenant_->id()=1001, group_id_=1, token=4) [2025-01-08 00:52:11.941528] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=77] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:52:11.943076] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D6-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:52:11.943163] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C2D6-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326331889252000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326331889252000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:11.945419] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:11.947164] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=48] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=0, clean_num=31457) [2025-01-08 00:52:11.947692] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46390][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5FC-0-0] [lt=22][errcode=-5024] REACH SYSLOG RATE LIMIT [bandwidth] [2025-01-08 00:52:11.947770] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46390][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5FC-0-0] [lt=0][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55814} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.950075] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:52:11.950391] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5FC-0-0] [lt=119] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7702}, to_seq_no={branch:0, seq:3130}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55814} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.950503] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=47] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:52:11.953066] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58700][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=46] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:52:11.965403] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58696][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3CAF-0-0] [lt=26] init point thread id with(&point=0x5572d50e7c80, idx_=1357, point=[thread id=58696, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58696) [2025-01-08 00:52:11.967508] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58710][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC3017-0-0] [lt=898] init point thread id with(&point=0x5572d50e7ec0, idx_=1366, point=[thread id=58710, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58710) [2025-01-08 00:52:11.973364] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=81] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:11.974751] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:11.974854] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326326571484000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:11.974882] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326326571484000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326326571484000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:52:11.977928] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C5FE-0-0] [lt=0][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55815} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.979760] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C5FE-0-0] [lt=41] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:11465}, to_seq_no={branch:0, seq:2300}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55815} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:11.994738] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:11.996861] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D1729C600-0-0] [lt=1][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.138", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:55816} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:52:11.999385] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46506][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1729C600-0-0] [lt=23] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6845}, to_seq_no={branch:0, seq:4425}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:55816} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:52:12.024994] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [46106][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=136][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5024, dropped:89, tid:46492}]) [2025-01-08 00:52:12.049120] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=30, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:52:12.049183] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1098, avg_group_log_size=122) [2025-01-08 00:52:12.049206] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1098) [2025-01-08 00:52:12.049294] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:52:12.053524] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=30531) [2025-01-08 00:52:12.054196] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:52:12.056125] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58709][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6ACC-0-0] [lt=128] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=48, avg_log_gen_to_freeze_time=160, avg_log_gen_to_submit_time=271, avg_log_submit_to_flush_time=2872, avg_log_submit_to_slide_time=4678) [2025-01-08 00:52:12.057946] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=30531, hit_cnt=49, read_cnt=49, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:12.069688] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8A-00062B2D1BC9C7F6-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1736326332067039000, start_id=1736326332067039000, end_id=1736326332067039001) [2025-01-08 00:52:12.082512] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=109] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=89, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:52:12.082575] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:52:12.084472] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9272) [2025-01-08 00:52:12.093584] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784CD1-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.095121] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784CD1-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.095671] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=90] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:52:12.095702] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=21] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=88) [2025-01-08 00:52:12.096832] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:12.101442] INFO [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1165) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] refresh queuing mode(ret=0, tenant_id=1002, stream_cnt=0, update_schema_cnt=0, cost_time=16) [2025-01-08 00:52:12.101510] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=64] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1002, interval_step=1, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1002, bucket_cnt:0, tenant_mem_user_limit:3221225472, tenant_mem_user_hold:243437568, cached_node_array:0x2ad8ba632ee0}) [2025-01-08 00:52:12.115746] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=9273, avg_group_log_size=662) [2025-01-08 00:52:12.118947] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49D1-0-0] [lt=48] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1323, avg_group_log_size=120) [2025-01-08 00:52:12.118980] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49D1-0-0] [lt=31] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1323) [2025-01-08 00:52:12.122432] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1001, new_min_version={val:1736326331875983000, v:0}, new_max_version={val:1736326331875983000, v:0}, last_min_version={val:1736326331691762000, v:0}, last_max_version={val:1736326331691762000, v:0}, total_time_us=9927, delta_us=246398) [2025-01-08 00:52:12.122468] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1001, current_version={val:1736326331875983000, v:0}, min_version={val:1736326331875983000, v:0}, max_version={val:1736326331875983000, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:52:12.129860] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C71-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:52:12.141845] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:52:12.142471] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:52:12.143992] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221589416, support_ssl=false) [2025-01-08 00:52:12.144122] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=36] sm conn init succ(conn.sessid_=3221589416, sess.client_addr_="127.0.0.1:51780") [2025-01-08 00:52:12.144144] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x2ad8efffadb0, session_id:3221589416, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:156}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.144181] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221589417, support_ssl=false) [2025-01-08 00:52:12.144285] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=72] sm conn init succ(conn.sessid_=3221589417, sess.client_addr_="127.0.0.1:51782") [2025-01-08 00:52:12.144300] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x2ad8efff8030, session_id:3221589417, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:192}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.144328] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221589418, support_ssl=false) [2025-01-08 00:52:12.144359] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221589418, sess.client_addr_="127.0.0.1:51786") [2025-01-08 00:52:12.144422] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x2ad8efff8db0, session_id:3221589418, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:193}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.145567] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.145622] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=25] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.145648] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.145671] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.145689] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.145708] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46224][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.145791] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=48] new mysql sessid created(conn.sessid_=3221589419, support_ssl=false) [2025-01-08 00:52:12.145847] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221589419, sess.client_addr_="127.0.0.1:51772") [2025-01-08 00:52:12.145863] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x2ad8efffc030, session_id:3221589419, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:194}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.145892] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221589420, support_ssl=false) [2025-01-08 00:52:12.145923] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221589420, sess.client_addr_="127.0.0.1:51778") [2025-01-08 00:52:12.145926] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A79-0-0] [lt=52] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.145935] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8efffcdb0, session_id:3221589420, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:195}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.145958] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221589421, support_ssl=false) [2025-01-08 00:52:12.145986] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221589421, sess.client_addr_="127.0.0.1:51792") [2025-01-08 00:52:12.145997] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8ce1fe030, session_id:3221589421, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:196}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.146075] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=160] new mysql sessid created(conn.sessid_=3221589422, support_ssl=false) [2025-01-08 00:52:12.146117] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221589422, sess.client_addr_="127.0.0.1:51768") [2025-01-08 00:52:12.146130] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8a0dfc030, session_id:3221589422, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:197}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.146154] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221589423, support_ssl=false) [2025-01-08 00:52:12.146182] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221589423, sess.client_addr_="127.0.0.1:51770") [2025-01-08 00:52:12.146260] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=73] accept one succ(*s={this:0x2ad8bb804db0, session_id:3221589423, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:198}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.146317] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221589424, support_ssl=false) [2025-01-08 00:52:12.146356] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221589424, sess.client_addr_="127.0.0.1:51774") [2025-01-08 00:52:12.146369] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8bb806030, session_id:3221589424, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:200}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.146392] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221589425, support_ssl=false) [2025-01-08 00:52:12.146421] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221589425, sess.client_addr_="127.0.0.1:51784") [2025-01-08 00:52:12.146433] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8bb806db0, session_id:3221589425, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:201}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.146455] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221589426, support_ssl=false) [2025-01-08 00:52:12.146482] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221589426, sess.client_addr_="127.0.0.1:51788") [2025-01-08 00:52:12.146493] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x2ad8bb808030, session_id:3221589426, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:202}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.146542] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221589427, support_ssl=false) [2025-01-08 00:52:12.146579] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221589427, sess.client_addr_="127.0.0.1:51776") [2025-01-08 00:52:12.146591] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8bb808db0, session_id:3221589427, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:203}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.146633] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:122) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221589428, support_ssl=false) [2025-01-08 00:52:12.146667] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:142) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221589428, sess.client_addr_="127.0.0.1:51790") [2025-01-08 00:52:12.146679] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1103) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x2ad8bb80a030, session_id:3221589428, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:204}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.147982] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A79-0-0] [lt=18] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589418, proxy_sessid=0, client_sessid=3221589418, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.148147] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A7A-0-0] [lt=30] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.149368] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A7A-0-0] [lt=15] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589417, proxy_sessid=0, client_sessid=3221589417, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.149562] INFO load_privilege_info (obmp_connect.cpp:582) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A7B-0-0] [lt=27] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.150404] INFO [SERVER] process (obmp_connect.cpp:523) [46335][T1_MysqlQueueTh][T1][Y0-00062B2D1AF84A7B-0-0] [lt=14] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589416, proxy_sessid=0, client_sessid=3221589416, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.151132] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46494][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CF0-0-0] [lt=12] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12276, add_timestamp:1736326332151114}) [2025-01-08 00:52:12.151997] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C0-0-0] [lt=97] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12276, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:12.153662] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6724874240, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:12.162752] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=11] gc stale ls task succ [2025-01-08 00:52:12.163666] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.163746] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.163771] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.163795] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46223][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.163825] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.163854] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.163870] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.163888] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.163902] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.163917] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.163931] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.163948] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.163961] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.163980] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46225][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.164018] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=31] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1002}, {id:1001}]) [2025-01-08 00:52:12.164071] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=146] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:12.164090] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-01-08 00:52:12.164099] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:12.164112] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:12.164137] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193836077255, flush_io_finished_ret_:0, flush_io_finished_round_:3818, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:12.164369] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8E-0-0] [lt=26] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.165328] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8E-0-0] [lt=18] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589428, proxy_sessid=0, client_sessid=3221589428, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.165422] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8F-0-0] [lt=28] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.166066] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A8F-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589427, proxy_sessid=0, client_sessid=3221589427, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.166169] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A90-0-0] [lt=44] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.166532] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857DF-0-0] [lt=1] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11003, add_timestamp:1736326332166523}) [2025-01-08 00:52:12.166978] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A90-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589426, proxy_sessid=0, client_sessid=3221589426, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.167068] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A91-0-0] [lt=25] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.167146] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857E0-0-0] [lt=24] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11017, add_timestamp:1736326332167142}) [2025-01-08 00:52:12.167830] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A91-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589425, proxy_sessid=0, client_sessid=3221589425, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.167910] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A92-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.168634] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A92-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589424, proxy_sessid=0, client_sessid=3221589424, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.168713] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A93-0-0] [lt=22] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.169404] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A93-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589423, proxy_sessid=0, client_sessid=3221589423, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.169486] INFO load_privilege_info (obmp_connect.cpp:582) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A94-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.170122] INFO [SERVER] process (obmp_connect.cpp:523) [46334][T1_MysqlQueueTh][T1][Y0-00062B2D17784A94-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589422, proxy_sessid=0, client_sessid=3221589422, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.170439] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.170491] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.170513] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.170534] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.170550] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:83) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=ocp_monitor, tenant_name=) [2025-01-08 00:52:12.170569] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:287) [46226][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-01-08 00:52:12.170644] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8C-0-0] [lt=27] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.171438] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8C-0-0] [lt=17] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589421, proxy_sessid=0, client_sessid=3221589421, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.171530] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8D-0-0] [lt=26] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.172036] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857E0-0-0] [lt=17][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:52:12.172404] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46493][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B1857E1-0-0] [lt=18] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12276, add_timestamp:1736326332172399}) [2025-01-08 00:52:12.172558] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8D-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589420, proxy_sessid=0, client_sessid=3221589420, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.172757] INFO load_privilege_info (obmp_connect.cpp:582) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8E-0-0] [lt=23] no tenant name set, use default tenant name(tenant_name=sys) [2025-01-08 00:52:12.173654] INFO [SERVER] process (obmp_connect.cpp:523) [46333][T1_MysqlQueueTh][T1][Y0-00062B2D17584A8E-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=%, sessid=3221589419, proxy_sessid=0, client_sessid=3221589419, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-01-08 00:52:12.173736] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=97] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:12.173784] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:52:12.174984] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA4-0-0] [lt=11] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10008, add_timestamp:1736326332174979}) [2025-01-08 00:52:12.176616] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:12.176636] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:52:12.177017] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA5-0-0] [lt=142][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12276, add_timestamp:1736326332177013}) [2025-01-08 00:52:12.177036] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA5-0-0] [lt=18][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:12276, add_timestamp:1736326332177013}) [2025-01-08 00:52:12.177045] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA5-0-0] [lt=8][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=12276, vtable_locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317155336, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317155336, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317155336, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.178706] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA4-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.179186] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84ABD-0-0] [lt=18] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11008, add_timestamp:1736326332179182}) [2025-01-08 00:52:12.180566] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA5-0-0] [lt=30][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.180715] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B484BA5-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.181162] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84ABE-0-0] [lt=15] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11021, add_timestamp:1736326332181158}) [2025-01-08 00:52:12.184530] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84ABF-0-0] [lt=74] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12217, add_timestamp:1736326332184525}) [2025-01-08 00:52:12.184565] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84ABF-0-0] [lt=21] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12142, add_timestamp:1736326332184564}) [2025-01-08 00:52:12.184581] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84ABF-0-0] [lt=10] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:11035, add_timestamp:1736326332184580}) [2025-01-08 00:52:12.184595] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84ABF-0-0] [lt=9] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12072, add_timestamp:1736326332184595}) [2025-01-08 00:52:12.184609] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46498][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84ABF-0-0] [lt=8] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12068, add_timestamp:1736326332184608}) [2025-01-08 00:52:12.185507] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46508][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AC0-0-0] [lt=38][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326332185503}) [2025-01-08 00:52:12.185532] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46508][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AC0-0-0] [lt=25][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:11017, add_timestamp:1736326332185503}) [2025-01-08 00:52:12.185541] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46508][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AC0-0-0] [lt=8][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=11017, vtable_locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354955, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354955, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317354955, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.190792] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_event.cpp:103) [46508][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AC0-0-0] [lt=32][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:52:12.191616] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46382][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84AC1-0-0] [lt=75][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.192342] WDIAG [SHARE.LOCATION] add_update_task (ob_vtable_location_service.cpp:388) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CF3-0-0] [lt=87][errcode=-4023] fail to add task(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:10008, add_timestamp:1736326332192337}) [2025-01-08 00:52:12.192366] WDIAG [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:444) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CF3-0-0] [lt=24][errcode=-4023] add location update task failed(ret=-4023, ret="OB_EAGAIN", task={tenant_id:1, table_id:10008, add_timestamp:1736326332192337}) [2025-01-08 00:52:12.192376] WDIAG [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:156) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CF3-0-0] [lt=9][errcode=0] add location update task failed(temp_ret=-4023, temp_ret="OB_EAGAIN", tenant_id=1, table_id=10008, vtable_locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176070, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176070, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326317176070, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.192992] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46382][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CF4-0-0] [lt=19] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:52:12.193041] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46382][T1_L0_G0][T1001][YB42C0A89F89-00062B2D17884CF4-0-0] [lt=16] get_stable_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:52:12.193067] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46382][T1_L0_G0][T1002][YB42C0A89F89-00062B2D17884CF4-0-0] [lt=10] get_stable_disk_usage(ret=0, capacity(MB):=4608, used(MB):=280) [2025-01-08 00:52:12.195813] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46499][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CF3-0-0] [lt=33][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.196020] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46382][T1_L0_G0][T1][YB42C0A89F89-00062B2D17884CF4-0-0] [lt=8][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.196600] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=31] temporary file module use no blocks [2025-01-08 00:52:12.196623] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:12.196639] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:12.196653] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193836107793, flush_io_finished_ret_:0, flush_io_finished_round_:1990, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:12.198279] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:12.199067] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C0-0-0] [lt=19] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12276, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332199040, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332199040, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332199040, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.199110] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C0-0-0] [lt=42] success to process renew task(task={tenant_id:1, table_id:12276, add_timestamp:1736326332151114}, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332199040, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332199040, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332199040, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.199152] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C1-0-0] [lt=26] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11003, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:12.200412] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4015] clock out of order(cur_ts=1736326332200410, cur_ts_=1736326332165962, delta=34448) [2025-01-08 00:52:12.205611] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=11] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:52:12.206964] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:52:12.207073] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=22] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326325712447000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:52:12.213549] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49D3-0-0] [lt=33] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:52:12.213629] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49D3-0-0] [lt=38] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:52:12.213642] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A49D3-0-0] [lt=12] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22562, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:52:12.213855] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=7] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=179, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:52:12.213878] INFO [LIB] stat (utility.h:1210) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=22] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=51, stat_interval=1000000, avg (count/cost)=10, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:52:12.215315] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2632, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:52:12.215378] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=60] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1323) [2025-01-08 00:52:12.215403] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=292, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:52:12.215475] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=111] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=64, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:52:12.216495] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918545E-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.217111] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58641][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BE2-0-0] [lt=67] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=31, avg_log_submit_to_flush_time=2999, avg_log_submit_to_slide_time=20802) [2025-01-08 00:52:12.218449] INFO [PALF] read (log_cache.cpp:103) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=51] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1323, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:12.218486] INFO [PALF] next (palf_iterator.h:183) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18531, miss_cnt:0, cache_read_size:2266919, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:12.219667] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C1-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11003, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332219640, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332219640, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332219640, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.219784] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C1-0-0] [lt=114] success to process renew task(task={tenant_id:1, table_id:11003, add_timestamp:1736326332166523}, locations=[{table_id:11003, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332219640, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332219640, sql_renew_time:0, is_mark_fail:false}, {table_id:11003, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332219640, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.219884] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C2-0-0] [lt=80] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11017, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:12.221562] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50897][T1002_L0_G2][T1002][YB42C0A89F8A-00062B2D1E79E607-0-0] [lt=24] get_min_unreplayed_log_info(lsn={lsn:2266919}, scn={val:1736326331889252004, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326332217088, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2266919}, next_to_submit_scn_:{val:1736326331889252004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2266797}, end_lsn:{lsn:2266919}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2266919}, readable_log_tail:{lsn:2266919}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64837849, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326331889252003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326331889252003, v:0}, data_checksum:3348837161, flag:57066}}, init_mode_version:0, accumulate_checksum:744609951, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326331889252003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18531, miss_cnt:0, cache_read_size:2266919, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:12.223902] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50625][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49D0F2-0-0] [lt=33][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.226154] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C2-0-0] [lt=59] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11017, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226134, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226134, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226134, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.226257] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C2-0-0] [lt=100] success to process renew task(task={tenant_id:1, table_id:11017, add_timestamp:1736326332167142}, locations=[{table_id:11017, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226134, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226134, sql_renew_time:0, is_mark_fail:false}, {table_id:11017, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226134, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.226323] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C3-0-0] [lt=28] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12276, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:12.226672] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C3-0-0] [lt=15] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12276, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226661, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226661, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226661, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.226708] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C3-0-0] [lt=35] success to process renew task(task={tenant_id:1, table_id:12276, add_timestamp:1736326332172399}, locations=[{table_id:12276, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226661, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226661, sql_renew_time:0, is_mark_fail:false}, {table_id:12276, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332226661, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.226740] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C4-0-0] [lt=24] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=10008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:12.228768] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1736326332226112000, start_id=1736326332226112000, end_id=1736326332226112001) [2025-01-08 00:52:12.232074] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8ce1fe030, session_id:3221589421, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232123] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=41] kill and revert session(conn.sessid_=3221589421, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.232135] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffcdb0, session_id:3221589420, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232152] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221589420, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.232158] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffc030, session_id:3221589419, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232173] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] kill and revert session(conn.sessid_=3221589419, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.232268] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=92] sock ref clean, do destroy(*s={this:0x2ad8ce1fe030, session_id:3221589421, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:196}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:667, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232310] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221589421, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.232321] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221589421, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CF4-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:562, rec_:562, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.232374] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=50] sock ref clean, do destroy(*s={this:0x2ad8efffcdb0, session_id:3221589420, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:195}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:541, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232387] INFO [SERVER] run (obmp_disconnect.cpp:83) [46379][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=31] free session successfully(sessid=3221589421) [2025-01-08 00:52:12.232407] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221589420, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.232411] INFO [SERVER] run (obmp_disconnect.cpp:90) [46379][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=23] mark session id unused(sessid=3221589421) [2025-01-08 00:52:12.232416] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221589420, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CF3-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:436, rec_:436, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.232444] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] sock ref clean, do destroy(*s={this:0x2ad8efffc030, session_id:3221589419, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:194}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:239, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232458] INFO [SERVER] run (obmp_disconnect.cpp:83) [46379][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=7] free session successfully(sessid=3221589420) [2025-01-08 00:52:12.232469] INFO [SERVER] run (obmp_disconnect.cpp:90) [46379][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221589420) [2025-01-08 00:52:12.232473] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221589419, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.232481] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46226][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221589419, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D17884CF2-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:134, rec_:134, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:7}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:8}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.232512] INFO [SERVER] run (obmp_disconnect.cpp:83) [46379][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221589419) [2025-01-08 00:52:12.232522] INFO [SERVER] run (obmp_disconnect.cpp:90) [46379][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=10] mark session id unused(sessid=3221589419) [2025-01-08 00:52:12.232521] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8bb808030, session_id:3221589426, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232548] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221589426, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.232555] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8bb806db0, session_id:3221589425, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232570] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221589425, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.232576] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8a0dfc030, session_id:3221589422, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232591] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221589422, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.232597] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] sock ref clean, do destroy(*s={this:0x2ad8bb808030, session_id:3221589426, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:202}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:738, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232625] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221589426, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.232633] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221589426, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AC1-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:633, rec_:633, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.232662] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] sock ref clean, do destroy(*s={this:0x2ad8bb806db0, session_id:3221589425, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:201}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:654, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232689] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221589425, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.232696] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221589425, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84AC0-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:549, rec_:549, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:12, seq_:19}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:20}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.232706] INFO [SERVER] run (obmp_disconnect.cpp:83) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=106] free session successfully(sessid=3221589426) [2025-01-08 00:52:12.232723] INFO [SERVER] run (obmp_disconnect.cpp:90) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=16] mark session id unused(sessid=3221589426) [2025-01-08 00:52:12.232722] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] sock ref clean, do destroy(*s={this:0x2ad8a0dfc030, session_id:3221589422, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:197}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:238, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232749] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221589422, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.232764] INFO [SERVER] run (obmp_disconnect.cpp:83) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221589425) [2025-01-08 00:52:12.232773] INFO [SERVER] run (obmp_disconnect.cpp:90) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=9] mark session id unused(sessid=3221589425) [2025-01-08 00:52:12.232756] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221589422, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84ABD-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:133, rec_:133, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:31, seq_:37}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:38}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.232802] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8bb80a030, session_id:3221589428, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232827] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221589428, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.232835] INFO [SERVER] run (obmp_disconnect.cpp:83) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221589422) [2025-01-08 00:52:12.232834] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8bb808db0, session_id:3221589427, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232844] INFO [SERVER] run (obmp_disconnect.cpp:90) [46507][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=8] mark session id unused(sessid=3221589422) [2025-01-08 00:52:12.232848] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] kill and revert session(conn.sessid_=3221589427, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.232854] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8bb80a030, session_id:3221589428, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:204}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232884] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221589428, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.232892] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221589428, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484BA5-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.232920] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] sock ref clean, do destroy(*s={this:0x2ad8bb808db0, session_id:3221589427, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:203}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:564, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.232934] INFO [SERVER] run (obmp_disconnect.cpp:83) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=19] free session successfully(sessid=3221589428) [2025-01-08 00:52:12.232945] INFO [SERVER] run (obmp_disconnect.cpp:90) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=11] mark session id unused(sessid=3221589428) [2025-01-08 00:52:12.232947] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221589427, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.232955] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46223][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221589427, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B484BA4-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:459, rec_:459, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:2, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.232985] INFO [SERVER] run (obmp_disconnect.cpp:83) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221589427) [2025-01-08 00:52:12.232994] INFO [SERVER] run (obmp_disconnect.cpp:90) [46495][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=8] mark session id unused(sessid=3221589427) [2025-01-08 00:52:12.233343] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=18] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):144, sequence:2}) [2025-01-08 00:52:12.233757] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C4-0-0] [lt=9] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10008, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332233733, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332233733, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332233733, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.233795] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C4-0-0] [lt=36] success to process renew task(task={tenant_id:1, table_id:10008, add_timestamp:1736326332174979}, locations=[{table_id:10008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332233733, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332233733, sql_renew_time:0, is_mark_fail:false}, {table_id:10008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332233733, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.233833] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C5-0-0] [lt=25] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11008, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:12.234045] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8db0, session_id:3221589418, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.234074] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221589418, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.234081] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efff8030, session_id:3221589417, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.234250] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=161] kill and revert session(conn.sessid_=3221589417, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.234262] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8efffadb0, session_id:3221589416, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.234279] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221589416, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.234286] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2ad8efff8db0, session_id:3221589418, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:193}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:473, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.234317] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3221589418, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.234327] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221589418, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857E1-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:368, rec_:368, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.234358] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=30] sock ref clean, do destroy(*s={this:0x2ad8efff8030, session_id:3221589417, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:192}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:346, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.234480] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=40] unlock session of tenant(conn.sessid_=3221589417, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.234455] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=42] free session successfully(sessid=3221589418) [2025-01-08 00:52:12.234506] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=50] mark session id unused(sessid=3221589418) [2025-01-08 00:52:12.234493] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221589417, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857E0-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:241, rec_:241, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.234523] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] sock ref clean, do destroy(*s={this:0x2ad8efffadb0, session_id:3221589416, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:156}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:203, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.234585] INFO [SERVER] run (obmp_disconnect.cpp:83) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=8] free session successfully(sessid=3221589417) [2025-01-08 00:52:12.234598] INFO [SERVER] run (obmp_disconnect.cpp:90) [46501][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=12] mark session id unused(sessid=3221589417) [2025-01-08 00:52:12.234612] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221589416, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.234624] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46224][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221589416, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1B1857DF-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:98, rec_:98, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:10}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:11}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.234724] INFO [SERVER] run (obmp_disconnect.cpp:83) [46503][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=32] free session successfully(sessid=3221589416) [2025-01-08 00:52:12.234741] INFO [SERVER] run (obmp_disconnect.cpp:90) [46503][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=17] mark session id unused(sessid=3221589416) [2025-01-08 00:52:12.239869] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C5-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11008, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332239851, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332239851, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332239851, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.239911] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C5-0-0] [lt=41] success to process renew task(task={tenant_id:1, table_id:11008, add_timestamp:1736326332179182}, locations=[{table_id:11008, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332239851, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332239851, sql_renew_time:0, is_mark_fail:false}, {table_id:11008, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332239851, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.239946] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C6-0-0] [lt=25] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=11021, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:52:12.241663] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB871D8-0-0] [lt=6][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:52:12.241708] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:52:12.241719] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:52:12.243947] WDIAG [SERVER] inner_get_next_row (ob_all_virtual_sys_stat.cpp:301) [46506][T1_L0_G0][T1][YB42C0A89F89-00062B2D1AE84ABE-0-0] [lt=114][errcode=-4008] execute fail(ret=-4008) [2025-01-08 00:52:12.247422] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C6-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11021, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247398, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247398, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247398, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.247471] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C6-0-0] [lt=48] success to process renew task(task={tenant_id:1, table_id:11021, add_timestamp:1736326332181158}, locations=[{table_id:11021, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247398, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247398, sql_renew_time:0, is_mark_fail:false}, {table_id:11021, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247398, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:52:12.247519] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C7-0-0] [lt=27] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12217, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247512, sql_renew_time:1736326332247512, is_mark_fail:false}]) [2025-01-08 00:52:12.247535] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C7-0-0] [lt=15] success to process renew task(task={tenant_id:1, table_id:12217, add_timestamp:1736326332184525}, locations=[{table_id:12217, partition_id:12217, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247512, sql_renew_time:1736326332247512, is_mark_fail:false}]) [2025-01-08 00:52:12.247556] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C8-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12142, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247551, sql_renew_time:1736326332247551, is_mark_fail:false}]) [2025-01-08 00:52:12.247570] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C8-0-0] [lt=14] success to process renew task(task={tenant_id:1, table_id:12142, add_timestamp:1736326332184564}, locations=[{table_id:12142, partition_id:12142, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247551, sql_renew_time:1736326332247551, is_mark_fail:false}]) [2025-01-08 00:52:12.247590] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C9-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=11035, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247585, sql_renew_time:1736326332247585, is_mark_fail:false}]) [2025-01-08 00:52:12.247603] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859C9-0-0] [lt=13] success to process renew task(task={tenant_id:1, table_id:11035, add_timestamp:1736326332184580}, locations=[{table_id:11035, partition_id:11035, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247585, sql_renew_time:1736326332247585, is_mark_fail:false}]) [2025-01-08 00:52:12.247623] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859CA-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12072, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247618, sql_renew_time:1736326332247618, is_mark_fail:false}]) [2025-01-08 00:52:12.247636] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859CA-0-0] [lt=13] success to process renew task(task={tenant_id:1, table_id:12072, add_timestamp:1736326332184595}, locations=[{table_id:12072, partition_id:12072, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247618, sql_renew_time:1736326332247618, is_mark_fail:false}]) [2025-01-08 00:52:12.247658] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859CB-0-0] [lt=12] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12068, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247653, sql_renew_time:1736326332247653, is_mark_fail:false}]) [2025-01-08 00:52:12.247671] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859CB-0-0] [lt=13] success to process renew task(task={tenant_id:1, table_id:12068, add_timestamp:1736326332184608}, locations=[{table_id:12068, partition_id:12068, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332247653, sql_renew_time:1736326332247653, is_mark_fail:false}]) [2025-01-08 00:52:12.248429] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8bb804db0, session_id:3221589423, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.248484] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=47] kill and revert session(conn.sessid_=3221589423, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.248498] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] sock ref clean, do destroy(*s={this:0x2ad8bb804db0, session_id:3221589423, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:198}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:793, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.248534] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221589423, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.248545] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221589423, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84ABE-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:688, rec_:688, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:3, seq_:5}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:219, seq_:225}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:226}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.248856] INFO [SERVER] run (obmp_disconnect.cpp:83) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=56] free session successfully(sessid=3221589423) [2025-01-08 00:52:12.248892] INFO [SERVER] run (obmp_disconnect.cpp:90) [46492][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=34] mark session id unused(sessid=3221589423) [2025-01-08 00:52:12.250723] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=80, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:52:12.250759] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:52:12.251400] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=27] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:2, threads_idle_time:5345953020, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:52:12.253342] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=2229, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:52:12.253446] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=102] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1944) [2025-01-08 00:52:12.253465] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=1638, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:52:12.253499] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1944) [2025-01-08 00:52:12.253517] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=32, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:52:12.254015] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=251] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=3919, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:52:12.254094] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1098) [2025-01-08 00:52:12.254116] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=269, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:52:12.254204] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=30, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:52:12.262952] INFO [LIB] stat (utility.h:1210) [58640][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BE4-0-0] [lt=24] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=12, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:52:12.262980] INFO [LIB] stat (utility.h:1210) [58640][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8BE4-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=6951, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:52:12.263085] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB APPEND COST TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=75, this=0x2ad934926280, extra_info="") [2025-01-08 00:52:12.263110] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=25] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=5516, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:52:12.263168] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=57] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:52:12.263303] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58640][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A9819-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=107, avg_log_submit_to_flush_time=4060, avg_log_submit_to_slide_time=6235) [2025-01-08 00:52:12.263176] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=-961, this=0x2ad934926910, extra_info="") [2025-01-08 00:52:12.263329] INFO [LIB] stat (utility.h:1210) [50476][T1002_ApplySrv1][T1002][Y0-0000000000000000-0-0] [lt=151] [APPLY STAT CB TOTAL TIME](cur_stat_count=22, stat_interval=5000000, avg (count/cost)=4630, this=0x2ad934926b40, extra_info="") [2025-01-08 00:52:12.263359] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:12.270354] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58716][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:12.271176] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46115][IO_SYNC_CH0][T0][Y0-0000000000000000-0-0] [lt=118] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:5345953020, tenant_id:500}, cur_thread_count=2, cnt=1, new_thread_count=3) [2025-01-08 00:52:12.271427] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:52:12.272092] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84562758}], scn_array=[cnt:1, {val:1736326332056064001, v:0}], curr_size=122, accum_size=708, time_cost=21090) [2025-01-08 00:52:12.272280] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=41, avg_log_batch_cnt=1, total_group_log_size=25531, avg_group_log_size=622) [2025-01-08 00:52:12.281740] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] ====== traversal_flush timer task ====== [2025-01-08 00:52:12.281839] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185417-0-0] [lt=27] get_max_applied_scn(scn={val:1736326332067039000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18759095}, palf_committed_end_scn_:{val:1736326332067039001, v:0}, last_check_scn_:{val:1736326332067039001, v:0}, max_applied_cb_scn_:{val:1736326332067039000, v:0}}) [2025-01-08 00:52:12.281869] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185417-0-0] [lt=26] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18759095}, readable_log_tail:{lsn:18759095}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48345673, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:52:12.281951] INFO [MDS] flush (mds_table_mgr.cpp:195) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185417-0-0] [lt=76] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326332067039000, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326329353096003, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326340087570000, latest_log_ts:{val:1736326323734680001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:12.282065] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185417-0-0] [lt=81] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326327857786003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326329353096003, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:12.282147] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185417-0-0] [lt=55] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326327876795004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326329353096003, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:12.282290] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185417-0-0] [lt=111] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326327851004004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326329353096003, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:12.282351] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185417-0-0] [lt=58] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:52:12.285464] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1022) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2ad8bb806030, session_id:3221589424, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.285549] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:272) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=77] kill and revert session(conn.sessid_=3221589424, proxy_sessid=0, ret=0) [2025-01-08 00:52:12.285561] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:997) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] sock ref clean, do destroy(*s={this:0x2ad8bb806030, session_id:3221589424, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:200}, err:5, last_decode_time:1736326332165962, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:224, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-01-08 00:52:12.285595] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:160) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] unlock session of tenant(conn.sessid_=3221589424, proxy_sessid=0, conn.tenant_id_=1) [2025-01-08 00:52:12.285607] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:248) [46225][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221589424, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A89F89-00062B2D1AE84ABF-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:7, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:119, rec_:119, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:2, seq_:4}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:5}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:3, seq_:8}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-01-08 00:52:12.285749] INFO [SERVER] run (obmp_disconnect.cpp:83) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=100] free session successfully(sessid=3221589424) [2025-01-08 00:52:12.285771] INFO [SERVER] run (obmp_disconnect.cpp:90) [46491][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=21] mark session id unused(sessid=3221589424) [2025-01-08 00:52:12.286550] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=133] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:52:12.290685] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] ====== check clog disk timer task ====== [2025-01-08 00:52:12.290736] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=44] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:52:12.292828] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A89-0-0] [lt=15] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326332292790}) [2025-01-08 00:52:12.292915] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859CC-0-0] [lt=11] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332292892, sql_renew_time:1736326332292892, is_mark_fail:false}]) [2025-01-08 00:52:12.292952] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A2859CC-0-0] [lt=36] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326332292790}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326332292892, sql_renew_time:1736326332292892, is_mark_fail:false}]) [2025-01-08 00:52:12.298394] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:12.300389] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=24] LogLoopThread round_cost_time(us)(round_cost_time=34) [2025-01-08 00:52:12.313834] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30] ls blacklist refresh finish(cost_time=22373) [2025-01-08 00:52:12.316602] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:52:12.321566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=192] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:12.321808] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:12.330151] INFO [LIB] stat (utility.h:1210) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C75-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:52:12.330370] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58704][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3C75-0-0] [lt=79] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:52:12.331661] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2714, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:52:12.331714] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1395, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:52:12.331888] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=60, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:52:12.342058] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=51] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:52:12.344747] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:12.354064] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:12.354827] INFO [LIB] stat (utility.h:1210) [58705][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A981C-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=41, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:52:12.354867] INFO [LIB] stat (utility.h:1210) [58705][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A981C-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=6290, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:52:12.355039] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58716][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A981D-0-0] [lt=39] init point thread id with(&point=0x5572d50e8040, idx_=1372, point=[thread id=58716, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58716) [2025-01-08 00:52:12.355067] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1944, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:12.362347] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8B-00062B2D183BE9A8-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326332361743000, start_id=1736326332361743000, end_id=1736326332361743001) [2025-01-08 00:52:12.365540] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=18717696) [2025-01-08 00:52:12.370338] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1736326332369882000, start_id=1736326332369882000, end_id=1736326332369882001) [2025-01-08 00:52:12.373051] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C89C2C9-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1736326332371014000, start_id=1736326332371014000, end_id=1736326332371014001) [2025-01-08 00:52:12.374927] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2574, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:52:12.374972] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=42] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10408) [2025-01-08 00:52:12.374996] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=438, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:52:12.375031] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=26, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:52:12.376741] INFO [LIB] stat (utility.h:1210) [58710][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC301C-0-0] [lt=49] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=23, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:52:12.376766] INFO [LIB] stat (utility.h:1210) [58710][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC301C-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=6451, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:52:12.376776] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58710][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC301C-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=113, avg_log_submit_to_flush_time=3295, avg_log_submit_to_slide_time=6337) [2025-01-08 00:52:12.376907] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10408, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:12.388438] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=45] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:52:12.393590] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=30] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:52:12.394692] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=97] PNIO [ratelimit] time: 1736326332394689, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007566, add_bytes: 0 [2025-01-08 00:52:12.398467] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:12.420528] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=70711296) [2025-01-08 00:52:12.421881] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:52:12.432031] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:52:12.433745] INFO [LIB] stat (utility.h:1210) [58705][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBF8-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:52:12.433773] INFO [LIB] stat (utility.h:1210) [58705][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBF8-0-0] [lt=28] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29454, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:52:12.433785] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58705][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228DBF8-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=17, avg_log_submit_to_flush_time=4053, avg_log_submit_to_slide_time=29436) [2025-01-08 00:52:12.433970] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=97] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:52:12.434016] INFO [PALF] next (palf_iterator.h:183) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:18433, miss_cnt:0, cache_read_size:2254633, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:52:12.440869] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=24] ====== traversal_flush timer task ====== [2025-01-08 00:52:12.441024] INFO [MDS] flush (mds_table_mgr.cpp:195) [50558][T1001_TimerWK0][T1001][YB42C0A89F89-00062B2D1E484E5D-0-0] [lt=24] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326332426299000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326339284252000, latest_log_ts:{val:1736326322666668001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:52:12.441128] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50558][T1001_TimerWK0][T1001][YB42C0A89F89-00062B2D1E484E5D-0-0] [lt=97] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:52:12.450996] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=1944, avg_group_log_size=216) [2025-01-08 00:52:12.451032] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1944) [2025-01-08 00:52:12.453138] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=17768) [2025-01-08 00:52:12.456241] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=78] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:52:12.458939] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:52:12.458993] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4681, iops:28, bw:130092, [delay/us]:prepare:9, schedule:428, submit:905, rt:1686, total:3030) [2025-01-08 00:52:12.459009] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=7, iops=7, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:12.459067] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4681, avg_iops:28, avg_bw:130095, [delay/us]: prepare:9, schedule:428, submit:905, device:1686, total:3030) [2025-01-08 00:52:12.459080] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:52:12.459120] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:81311, [delay/us]:prepare:14, schedule:292, submit:340, rt:1335, total:1982) [2025-01-08 00:52:12.459130] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:12.459165] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:81313, [delay/us]: prepare:14, schedule:292, submit:340, device:1335, total:1982) [2025-01-08 00:52:12.459230] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=134] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:12.459176] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:52:12.459266] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16261, [delay/us]:prepare:6, schedule:75, submit:1, rt:1939, total:2022) [2025-01-08 00:52:12.459267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:12.459289] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4201, iops:39, bw:162617, [delay/us]:prepare:17, schedule:274, submit:409, rt:1731, total:2432) [2025-01-08 00:52:12.459300] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=9, iops=9, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:12.459332] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16262, [delay/us]: prepare:6, schedule:75, submit:1, device:1939, total:2022) [2025-01-08 00:52:12.459348] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4201, avg_iops:39, avg_bw:162629, [delay/us]: prepare:17, schedule:274, submit:409, device:1731, total:2432) [2025-01-08 00:52:12.459359] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:52:12.459379] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4065, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:2627, total:2639) [2025-01-08 00:52:12.459402] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:52:12.459429] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4065, [delay/us]: prepare:0, schedule:0, submit:600000000, device:2627, total:2639) [2025-01-08 00:52:12.459447] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:52:12.459462] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:12.459470] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:52:12.459478] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:52:12.459486] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=714kB/s, net_out=358kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:52:12.498612] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:12.508771] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:52:12.508829] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=52] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=140, cache_node_num=140) [2025-01-08 00:52:12.508894] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=61] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=140, cache_node_num=140) [2025-01-08 00:52:12.508931] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=34] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:12.511485] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=10] Dumping All Cache Objs(alloc_obj_list.count()=140, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:2081, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191674}, {obj_id:2082, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180041}, {obj_id:2083, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221102}, {obj_id:2084, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2085, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2086, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2087, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2088, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2089, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2090, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:52:12.511923] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=434] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:52:12.519806] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:12.542017] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:12.542267] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:52:12.543756] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DBD6-0-0] [lt=6][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193837457589, cache_obj->added_lc()=false, cache_obj->get_object_id()=13721, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:52:12.544021] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:52:12.551785] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:364K/84:226473 disk:364K/84:226473 t1m2:112K/25:412339 t1m2g0:112K/25:412339 t1002m2:180K/44:116089 t1002m2g0:180K/44:116089 t1001m2:72K/15:240488 t1001m2g0:72K/15:240488 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:52:12.554548] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:12.598744] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:12.618132] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068B016-0-0] [lt=94] need check schema(new_schema_version=1736326191428712, cached_tenant_schema_version=1736326191428712, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=update __all_weak_read_service set min_version=?, max_version=? where tenant_id = ? and level_id = ? and level_value = ? and min_version = ? and max_version = ?) [2025-01-08 00:52:12.618539] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068B016-0-0] [lt=38] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.794573879241943359e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=389, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.605895266636067618e+02) [2025-01-08 00:52:12.618954] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] submit log success(service_type_=0, last_id=1736326349284252000, limited_id=1736326349284252000) [2025-01-08 00:52:12.618971] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] submit log with lock success(service_type_=0, last_id=1736326349284252000, limited_id=1736326349284252000) [2025-01-08 00:52:12.623022] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=66] update id meta(ret=0, service_type=0, limited_id=1736326349284252000, latest_scn={val:1736326332618421001, v:0}, *this={tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326349284252000, latest_log_ts:{val:1736326332618421001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:52:12.623145] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=122] handle submit callback(service_type_=0, success=true, used_ts=4192, log_ts={val:1736326332618421001, v:0}, limited_id=1736326349284252000) [2025-01-08 00:52:12.623161] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=15] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:52:12.623224] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=62] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:52:12.624942] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=13] get number(ret=0, service_type_=0, range=1, base_id=1736326332624508000, start_id=1736326332624508000, end_id=1736326332624508001) [2025-01-08 00:52:12.638653] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=79] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736324352943733, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:52:12.652088] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=75] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=25, stat_interval=1000000, avg (count/cost)=85, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:52:12.652132] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND COST TIME](cur_stat_count=25, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:52:12.653637] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=88] [PALF STAT WRITE LOG COST TIME](cur_stat_count=25, stat_interval=1000000, avg (count/cost)=3072, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:52:12.655821] INFO [LIB] stat (utility.h:1210) [58707][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5E3A-0-0] [lt=84] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=25, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:52:12.655857] INFO [LIB] stat (utility.h:1210) [58707][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5E3A-0-0] [lt=35] [PALF STAT LOG LIFE TIME](cur_stat_count=25, stat_interval=1000000, avg (count/cost)=6565, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:52:12.657961] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58719][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:52:12.658333] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1, group_id_=1, token=6) [2025-01-08 00:52:12.662432] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=12675) [2025-01-08 00:52:12.662497] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:158) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=84] submit log success(service_type_=0, last_id=1736326349254996000, limited_id=1736326349254996000) [2025-01-08 00:52:12.662509] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] submit log with lock success(service_type_=0, last_id=1736326349254996000, limited_id=1736326349254996000) [2025-01-08 00:52:12.663527] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=184, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:52:12.663716] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=25, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:52:12.665147] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] update id meta(ret=0, service_type=0, limited_id=1736326349254996000, latest_scn={val:1736326332661651001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326349254996000, latest_log_ts:{val:1736326332661651001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-01-08 00:52:12.665675] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:190) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=525] handle submit callback(service_type_=0, success=true, used_ts=3178, log_ts={val:1736326332661651001, v:0}, limited_id=1736326349254996000) [2025-01-08 00:52:12.665697] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:564) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=21] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-01-08 00:52:12.665708] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:602) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=10] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-01-08 00:52:12.666922] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58719][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5E3C-0-0] [lt=52] init point thread id with(&point=0x5572d50e8140, idx_=1376, point=[thread id=58719, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58719) [2025-01-08 00:52:12.667468] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=40] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:52:12.698872] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=156][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:12.704672] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=30] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8885, offset=2525171}, end_cursor:ObLogCursor{file_id=1, log_id=8886, offset=2525476}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894dd4050, len:2061, log_data_len:305, seq:8885, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:12.705391] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=72] current monitor number(seq_=-1) [2025-01-08 00:52:12.705689] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E84-0-0] [lt=100] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:52:12.705775] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E84-0-0] [lt=20] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326332369882001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326332369882001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:12.712318] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=95] PNIO [ratelimit] time: 1736326332712315, bytes: 889588064, bw: 0.497561 MB/s, add_ts: 1000134, add_bytes: 521800 [2025-01-08 00:52:12.712845] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E84-0-0] [lt=36] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11248) [2025-01-08 00:52:12.718729] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=40] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8886, offset=2525476}, end_cursor:ObLogCursor{file_id=1, log_id=8887, offset=2525773}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894dcc050, len:1756, log_data_len:297, seq:8886, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:12.719399] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E84-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:52:12.719489] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E84-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326332369882002, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326332369882002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:12.731130] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8887, offset=2525773}, end_cursor:ObLogCursor{file_id=1, log_id=8888, offset=2526070}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894dd8050, len:1459, log_data_len:297, seq:8887, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:12.731594] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=45] ====== tenant freeze timer task ====== [2025-01-08 00:52:12.732975] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=210] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8888, offset=2526070}, end_cursor:ObLogCursor{file_id=1, log_id=8889, offset=2526367}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894dd0050, len:1162, log_data_len:297, seq:8888, flush_finish:false, flush_ret:0}) [2025-01-08 00:52:12.742466] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:52:12.752319] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085183-0-0] [lt=14] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:52:12.755057] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6735360000, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:52:12.798979] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:52:12.826352] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=48] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:52:12.843722] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:52:12.843923] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:52:12.845002] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=41] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:52:12.852099] INFO [SHARE] get_balance_job (ob_balance_job_table_operator.cpp:209) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464B-0-0] [lt=54] empty balance job(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=select time_to_usec(gmt_create) as start_time, time_to_usec(gmt_modified) as finish_time, * from __all_balance_job) [2025-01-08 00:52:12.854303] INFO [BALANCE] build (ob_transfer_partition_task.cpp:120) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464B-0-0] [lt=39] [TRANSFER_PARTITION] no transfer partition task(tenant_id_=1002) [2025-01-08 00:52:12.854325] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1010) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464B-0-0] [lt=18] no transfer partition job [2025-01-08 00:52:12.854338] INFO [BALANCE] transfer_partition_ (ob_tenant_balance_service.cpp:1040) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464B-0-0] [lt=8] [TENANT_BALANCE] finish transfer partition(ret=0, ret="OB_SUCCESS", job_cnt=0, enable transfer=true) [2025-01-08 00:52:12.856665] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=44] gc stale ls task succ [2025-01-08 00:52:12.857268] INFO [BALANCE] gather_ls_status_stat (ob_tenant_balance_service.cpp:284) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464B-0-0] [lt=11] [TENANT_BALANCE] LS no need balance(ls_status={tenant_id:1002, ls_id:{id:1}, ls_group_id:0, status:"NORMAL", unit_group_id:0, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-01-08 00:52:12.857329] INFO [BALANCE] ls_balance_ (ob_tenant_balance_service.cpp:493) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464B-0-0] [lt=48] [TENANT_BALANCE] finish ls balance(ret=0, ret="OB_SUCCESS", need_ls_balance=false, job={tenant_id:0, job_id:{id:-1}, job_type:{val:-1, job_type:"INVALID"}, job_status:{val:-1, job_status:"INVALID"}, primary_zone_num:0, unit_group_num:0, comment:, balance_strategy:}, tasks=[]) [2025-01-08 00:52:12.857364] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58697][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB62D-0-0] [lt=29] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:52:12.857361] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:157) [50394][T1002_TBalance][T1002][YB42C0A89F89-00062B2D24A8464B-0-0] [lt=25] [TENANT_BALANCE] finish one round(ret=0, ret="OB_SUCCESS", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=3, unit_group_array_=[{unit_group_id:1001, status:0}], ls_array_=[{tenant_id:1002, ls_id:{id:1001}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone1", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1002}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone2", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, {tenant_id:1002, ls_id:{id:1003}, ls_group_id:1001, status:"NORMAL", unit_group_id:1001, primary_zone:"zone3", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}], idle_time_us=10000000, last_partition_balance_time=1736324440080711, last_statistic_bg_stat_time=1736326192603087, last_statistic_schema_version=1736326192037600, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-01-08 00:52:12.857409] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58697][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB62D-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326332361743004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326332361743004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:52:12.857648] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685541-0-0] [lt=49] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-01-08 00:52:12.857671] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C685541-0-0] [lt=18] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-01-08 00:52:12.859954] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=137] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:52:12.859998] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=37] temporary file module use no blocks [2025-01-08 00:52:12.860009] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:52:12.860024] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:52:12.860038] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193836773633, flush_io_finished_ret_:0, flush_io_finished_round_:1995, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:52:12.860431] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] no need to update min/max version(new_version={val:1736326332458415000, v:0}, min_version={val:1736326332458415000, v:0}, max_version={val:1736326332458415000, v:0}) [2025-01-08 00:52:12.860540] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE9AA-0-0] [lt=35] handle gts request statistics(total_rt=2126, total_cnt=330, avg_rt=6.422960725075528288e+00, avg_cnt=3.300000000000000000e+01) [2025-01-08 00:52:12.863580] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE9AC-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1736326332862987000, start_id=1736326332862987000, end_id=1736326332862987001) [2025-01-08 00:52:12.866508] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=81] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=18717696) [2025-01-08 00:52:12.870060] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:52:12.873075] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=42] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1})